System and method for presenting image content on multiple depth planes by providing multiple intra-pupil parallax views

ABSTRACT

An augmented reality display system is configured to direct a plurality of parallactically-disparate intra-pupil images into a viewer&#39;s eye. The parallactically-disparate intra-pupil images provide different parallax views of a virtual object, and impinge on the pupil from different angles. In the aggregate, the wavefronts of light forming the images approximate a continuous divergent wavefront and provide selectable accommodation cues for the user, depending on the amount of parallax disparity between the intra-pupil images. The amount of parallax disparity is selected using a light source that outputs light for different images from different locations, with spatial differences in the locations of the light output providing differences in the paths that the light takes to the eye, which in turn provide different amounts of parallax disparity. Advantageously, the wavefront divergence, and the accommodation cue provided to the eye of the user, may be varied by appropriate selection of parallax disparity, which may be set by selecting the amount of spatial separation between the locations of light output.

PRIORITY CLAIM

This application is a continuation application of U.S. application Ser.No. 15/789,895, filed on Oct. 20, 2017, which claims the benefit ofpriority of U.S. Provisional Application No. 62/411,490, filed on Oct.21, 2016, which is incorporated herein by reference.

INCORPORATION BY REFERENCE

This application incorporates by reference the entirety of each of thefollowing patent applications: U.S. application Ser. No. 14/555,585filed on Nov. 27, 2014; U.S. application Ser. No. 14/690,401 filed onApr. 18, 2015; U.S. application Ser. No. 14/212,961 filed on Mar. 14,2014; U.S. application Ser. No. 14/331,218 filed on Jul. 14, 2014; U.S.application Ser. No. 15/072,290 filed on Mar. 16, 2016; and U.S.Provisional Application No. 62/156,809, filed on May 4, 2015.

BACKGROUND Field

The present disclosure relates to optical devices, including augmentedreality and virtual reality imaging and visualization systems.

Description of the Related Art

Modern computing and display technologies have facilitated thedevelopment of systems for so called “virtual reality” or “augmentedreality” experiences, wherein digitally reproduced images or portionsthereof are presented to a user in a manner wherein they seem to be, ormay be perceived as, real. A virtual reality, or “VR”, scenariotypically involves presentation of digital or virtual image informationwithout transparency to other actual real-world visual input; anaugmented reality, or “AR”, scenario typically involves presentation ofdigital or virtual image information as an augmentation to visualizationof the actual world around the user. A mixed reality, or “MR”, scenariois a type of AR scenario and typically involves virtual objects that areintegrated into, and responsive to, the natural world. For example, inan MR scenario, AR image content may be blocked by or otherwise beperceived as interacting with objects in the real world.

Referring to FIG. 1 , an augmented reality scene 10 is depicted whereina user of an AR technology sees a real-world park-like setting 20featuring people, trees, buildings in the background, and a concreteplatform 30. In addition to these items, the user of the AR technologyalso perceives that he “sees” “virtual content” such as a robot statue40 standing upon the real-world platform 30, and a cartoon-like avatarcharacter 50 flying by, which seems to be a personification of a bumblebee, even though these elements 40, 50 do not exist in the real world.Because the human visual perception system is complex, it is challengingto produce an AR technology that facilitates a comfortable,natural-feeling, rich presentation of virtual image elements amongstother virtual or real-world imagery elements.

Systems and methods disclosed herein address various challenges relatedto AR and VR technology.

SUMMARY

In some embodiments, a head-mounted display system is provided. Thedisplay system comprises a frame configured to mount on a viewer; alight source; a spatial light modulator configured to modulate lightfrom the light source; and projection optics mounted on the frame andconfigured to direct light from the spatial light modulator into an eyeof a viewer. The display system is configured to display a virtualobject on a depth plane by injecting a set of parallactically-disparateintra-pupil images of the object into the eye.

In some other embodiments, a method is provided for displaying imagecontent. The method comprises providing a spatial light modulator;providing a light source configured to output light to the spatial lightmodulator from a plurality of different light output locations; anddisplaying a virtual object on a depth plane by temporally sequentiallyinjecting a set of parallactically-disparate intra-pupil images of thevirtual object into an eye of a viewer. Each of the intra-pupil imagesis formed by outputting light from the light source to the spatial lightmodulator, wherein the light is outputted from one or more associatedlight output locations of the light source; modulating the light withthe spatial light modulator to form an intra-pupil image correspondingto the one or more associated light output locations; and propagatingthe modulated light to the eye. The one or more associated light outputlocations for each intra-pupil image is distinct from the one or moreassociated light output locations for others of the intra-pupil images.

In yet other embodiments, a display system is provided. The displaysystem comprises a light source comprising a plurality of spatiallydistinct light output locations; a spatial light modulator configured tomodulate light from the light source; and projection optics mounted onthe frame and configured to direct light from the spatial lightmodulator into an eye of a viewer. The display system is configured todisplay a virtual object on a depth plane by temporally sequentiallyinjecting a set of parallactically-disparate intra-pupil images of theobject into the eye.

In some other embodiments, a method is provided for displaying imagecontent. The method comprises providing a head-mounted displaycomprising a light source and a spatial light modulator. The methodfurther comprises displaying a virtual object on a depth plane byinjecting, within a flicker fusion threshold, a set ofparallactically-disparate intra-pupil images of the virtual object fromthe display into an eye of a viewer.

In addition, various innovative aspects of the subject matter describedin this disclosure may be implemented in the following embodiments:

1. A method for displaying image content, the method comprising:

-   -   providing a spatial light modulator;    -   providing a light source configured to output light to the        spatial light modulator from a plurality of different light        output locations; and    -   displaying a virtual object on a depth plane by temporally        sequentially injecting a set of parallactically-disparate        intra-pupil images of the virtual object into an eye of a        viewer, wherein each of the intra-pupil images is formed by:        -   outputting light from the light source to the spatial light            modulator, wherein the light is outputted from one or more            associated light output locations of the light source;        -   modulating the light with the spatial light modulator to            form an intra-pupil image corresponding to the one or more            associated light output locations; and        -   propagating the modulated light to the eye,        -   wherein the one or more associated light output locations            for each intra-pupil image is distinct from the one or more            associated light output locations for others of the            intra-pupil images.

2. The method of Embodiment 1, wherein activating the one or moreassociated light-emitting regions comprises selecting the one or moreassociated light-emitting regions based upon the depth plane, wherein aphysical separation between light-emitting regions for the intra-pupilimages increases with decreasing distance of the depth plane to theviewer.

3. The method of any of Embodiments 1-2, wherein light rays forming eachof the parallactically-disparate image are collimated, wherein the depthplane is at less than optical infinity.

4. The method of any of Embodiments 1-3, wherein injecting the set ofparallactically-disparate intra-pupil images is conducted within atimeframe below the flicker fusion threshold of the viewer.

5. The method of Embodiment 4, wherein the flicker fusion threshold is1/60 of a second.

6. The method of any of Embodiments 1-5, further comprising an eyetracking sensor configured to track a gaze of the eye, whereindisplaying the virtual object comprises:

-   -   determining a gaze of the eye using the eye tracking sensor; and    -   selecting content for the intra-pupil images based upon the        determined gaze of the eye.

7. The method of any of Embodiments 1-6, further comprising projectionoptics configured to direct modulated light from the spatial lightmodulator to the eye.

8. The method of any of Embodiments 1-7, wherein the one or moreassociated light-emitting regions for the intra-pupil images partiallyoverlap.

9. The method of any of Embodiments 1-8, further comprising changing aposition of the one or more associated light-emitting regions duringinjection of at least one of the intra-pupil images into the eye.

10. A display system configured to perform the method of any ofEmbodiments 1-9.

11. A method for displaying image content, the method comprising:

-   -   providing a head-mounted display comprising:        -   a light source; and        -   a spatial light modulator; and    -   displaying a virtual object on a depth plane by injecting,        within a flicker fusion threshold, a set of        parallactically-disparate intra-pupil images of the virtual        object from the display into an eye of a viewer.

12. The method of Embodiment 11, wherein injecting the set ofparallactically-disparate intra-pupil images comprises temporallysequentially injecting individual ones of the intra-pupil images into aneye of a viewer.

13. The method of Embodiment 11, wherein injecting the set ofparallactically-disparate intra-pupil images comprises simultaneouslyinjecting multiple ones of the intra-pupil images.

14. The method of Embodiment 13, wherein injecting the set ofparallactically-disparate intra-pupil images comprises temporallysequentially injecting multiple intra-pupil images at a time.

15. The method of any of Embodiments 11-14, wherein the light beamsforming the intra-pupil images are collimated.

16. The method of any of Embodiments 11-14, wherein the light beamsforming the intra-pupil images have divergent wavefronts.

17. The method of any of Embodiments 11-16, wherein the light sourcecomprises a plurality of selectively activated light-emitting regions,wherein injecting the set of parallactically-disparate intra-pupilimages comprises activating a different light emitting region for eachintra-pupil image.

18. The method of any of Embodiments 11-17, wherein the light source isconfigured to output light from a plurality of distinct light outputlocations, further comprising jittering the light output locationsduring injection of at least one of the intra-pupil images into the eye.

19. A display system configured to perform the method of any ofEmbodiments 11-18.

20. A head-mounted display system comprising:

-   -   a frame configured to mount on a viewer;    -   a light source;    -   a spatial light modulator configured to modulate light from the        light source; and    -   projection optics mounted on the frame and configured to direct        light from the spatial light modulator into an eye of a viewer,    -   wherein the display system is configured to display a virtual        object on a depth plane by injecting a set of        parallactically-disparate intra-pupil images of the object into        the eye.

21. The display system of Embodiment 20, wherein the display system isconfigured to temporally multiplex display of individual intra-pupilimages.

22. The display system of any of Embodiments 20-21, wherein the displaysystem is configured to spatially multiplex display of the intra-pupilimages.

23. The display system of any of Embodiments 20-22, wherein the displaysystem is configured to temporally multiplex display of a plurality ofspatially-multiplexed intra-pupil images.

24. The display system of any of Embodiments 20-23, wherein theprojection optics comprises a waveguide comprising incoupling opticalelements and outcoupling optical elements.

25. The display system of Embodiment 24, wherein the projection opticscomprises a plurality of waveguides, wherein each waveguide isconfigured to output light of a different component color than otherwaveguides of the plurality of waveguides.

26. The display system of any of Embodiments 20-25, wherein the lightsource comprises a plurality of selectively-activated light-emittingregions.

27. The display system of Embodiment 26, wherein the light sourcecomprises at least one of a light-emitting diode array and a spatiallight modulator.

28. The display system of Embodiment 8, wherein the light-emitting diodearray comprises an organic light-emitting diode array or an inorganiclight-emitting diode array.

29. The display system of Embodiment 27, wherein the spatial lightmodulator light source comprises a liquid crystal array or a digitallight processing (DLP) chip.

30. The display system of any of Embodiments 20-29, wherein the displaysystem is configured to change a position of activated light-emittingregions during injection of at least one of the intra-pupil images intothe eye.

31. The display system of any of Embodiments 20-25, wherein the lightsource comprises:

-   -   a light emitter; and    -   an actuator configured to direct light to the spatial light        modulator along different paths.

32. The display system of Embodiment 31, wherein the actuator is adual-axis galvanometer.

33. The display system of Embodiment 31, wherein the light source is afiber scanner.

34. The display system of any of Embodiments 20-33, wherein the spatiallight modulator configured to modulate light from the light sourcecomprises an LCOS panel.

35. The display system of any of Embodiments 20-34, further comprisingan eye tracking sensor configured to track a gaze of the eye, whereinthe display system is configured to:

-   -   determine a gaze of the eye using the eye tracking sensor; and    -   select content for the intra-pupil images based upon the        determined gaze of the eye.

36. The display system of any of Embodiments 20-35, wherein the displaysystem is configured to synchronize a light output location of the lightsource with image content provided by the spatial light modulator.

37. The display system of any of Embodiments 20-36, further comprisingan optical mechanism between the spatial light modulator and theprojection optics, wherein the optical mechanism is configured to directlight from different locations of the spatial light modulator toprojection optics at different angles.

38. The display system of Embodiment 37, wherein the optical mechanismcomprises one or more of a prism or a lens structure.

39. The display system of Embodiment 38, wherein the lens structure is alenslet array.

40. A display system comprising:

-   -   a light source comprising a plurality of spatially distinct        light output locations;    -   a spatial light modulator configured to modulate light from the        light source; and    -   projection optics mounted on the frame and configured to direct        light from the spatial light modulator into an eye of a viewer,    -   wherein the display system is configured to display a virtual        object on a depth plane by temporally sequentially injecting a        set of parallactically-disparate intra-pupil images of the        object into the eye.

41. The display system of Embodiment 40, configured to output light fromdifferent light output locations of the light source for differentintra-pupil images.

42. The display system of Embodiment 41, configured to vary a lateralseparation between the light output locations based upon a distance ofthe depth plane from the eye of the viewer.

43. The display system of any of Embodiments 41-42, configured toincrease the lateral separation between light output locations withincreases in the distance of the depth plane from the eye of the viewer.

44. The display system of any of Embodiments 41-42, wherein the displaysystem is configured to change the light output locations duringinjection of at least one of the intra-pupil images into the eye.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a user's view of augmented reality (AR) through an ARdevice.

FIG. 2 illustrates a conventional display system for simulatingthree-dimensional imagery for a user.

FIG. 3 illustrates aspects of an approach for simulatingthree-dimensional imagery using multiple depth planes.

FIGS. 4A-4C illustrate relationships between curvature and focaldistance.

FIG. 5 illustrates an example of a waveguide stack for outputting imageinformation to a user.

FIG. 6A illustrates pre-accommodation and post-accommodation conditionsof an eye to a continuous incoming wavefront.

FIG. 6B illustrates pre-accommodation and post-accommodation conditionsof an eye to a piecewise approximation of a continuous incomingwavefront.

FIG. 7A illustrates an eye accommodating to a divergent wavefrontemanating from a finite focal-distance virtual image provided by aprojection system.

FIG. 7B illustrates a system for forming an approximation of thedivergent wavefront of FIG. 7A utilizing wavefront segments formed byinfinity-focused virtual images.

FIG. 8 illustrates examples of parallax views forming the divergentwavefront approximation of FIG. 7B.

FIG. 9 illustrates an example of a display system comprising aprojection system for forming the divergent wavefront approximation ofFIG. 7B.

FIG. 10 illustrates examples of sizes, shapes, and distributions forlight-emitting regions.

FIG. 11A illustrates another example of a projection system for formingthe divergent wavefront approximation of FIG. 7B.

FIG. 11B illustrates an example of a range of depth planes provided bythe projection system of FIG. 11A.

FIG. 12 illustrates an example of a light source configuration forprojection systems.

FIG. 13A illustrates an example of a projection system for placingvirtual objects on a default depth plane that is less than opticalinfinity.

FIG. 13B illustrates an example of a range of depth planes provided bythe projection system of FIG. 13A.

FIG. 14 illustrates an example of a projection system configured for thespatially multiplexed display of intra-pupil images.

FIG. 15 illustrates an example of a projection system configured forspatially and temporally multiplexed display of intra-pupil images.

FIG. 16 illustrates an example of a projection system comprising a pupilrelay combiner eyepiece for superimposing image content on a user's viewof the world.

FIG. 17 illustrates an example of a display system comprising an eyetracking system and a combiner eyepiece with a pupil expander.

FIG. 18 illustrates an example of a display system comprising an eyetracking system and a pupil rely combiner eyepiece with a pupil expanderconfigured to produce a non-infinity depth plane.

FIG. 19 illustrates a light source comprising mirrors for directing thepropagation of light to different light output locations.

FIG. 20 illustrates a light source comprising a fiber scanner.

FIG. 21 illustrates an example of an eyepiece comprising a stackedwaveguide assembly for outputting light of different wavelengthscorresponding to different component colors.

FIG. 22 illustrates an example of a wearable display system.

DETAILED DESCRIPTION

The human visual system may be made to perceive images presented by adisplay as being “3-dimensional” by providing slightly differentpresentations of the image to each of a viewer's left and right eyes.Depending on the images presented to each eye, the viewer perceives a“virtual” object in the images as being at a selected distance (e.g., ata certain “depth plane”) from the viewer. Simply providing differentpresentations of the image to the left and right eyes, however, maycause viewer discomfort. As discussed further herein, viewing comfortmay be increased by causing the eyes to accommodate to the imagessimilarly to the accommodation that would occur if the viewer wereviewing a real object at that depth plane on which the virtual object isplaced.

The proper accommodation for a virtual object on a given depth plane maybe elicited by presenting images to the eyes with light having awavefront divergence that matches the wavefront divergence of lightcoming from a real object on that depth plane. Some display systems usedistinct structures having distinct optical powers to provide theappropriate wavefront divergence. For example, one structure may providea specific amount of wavefront divergence (to place virtual objects onone depth plane) and another structure may provide a different amount ofwavefront divergence (to place virtual objects on a different depthplane). Thus, there may be a one-to-one correspondence between physicalstructures and the depth planes in these display systems. Due to theneed for a separate structure for each depth plane, such display systemsmay be bulky and/or heavy, which may be undesirable for someapplications, such as portable head-mounted displays. In addition, suchdisplay systems may be limited in the numbers of different accommodativeresponses they may elicit from the eyes, due to practical limits on thenumber of structures of different optical powers that may be utilized.

It has been found that a continuous wavefront, e.g. a continuousdivergent wavefront, may be approximated by injectingparallactically-disparate intra-pupil images directed into an eye. Insome embodiments, a display system may provide a range of accommodativeresponses without requiring a one-to-one correspondence between opticalstructures in the display and the accommodative response. For example,the same optical projection system may be utilized to output light witha selected amount of perceived wavefront divergence, corresponding to adesired depth plane, by injecting a set of parallactically-disparateintra-pupil images into the eye. These images may be referred to as“parallactically-disparate” intra-pupil images since each image may beconsidered to be a different parallax view of the same virtual object orscene, on a given depth plane. These are “intra-pupil” images since aset of images possessing parallax disparity is projected into the pupilof a single eye, e.g., the right eye of a viewer. Although some overlapmay occur, the light beams forming these images will have at least someareas without overlap and will impinge on the pupil from slightlydifferent angles. In some embodiments, the other eye of the viewer,e.g., the left eye, may be provided with its own set ofparallactically-disparate intra-pupil images. The sets ofparallactically-disparate intra-pupil images projected into each eye maybe slightly different, e.g., the images may show slightly differentviews of the same scene due to the slightly different perspectivesprovided by each eye.

The wavefronts of light forming each of the intra-pupil images projectedinto a pupil of an eye of a view, in the aggregate, may approximate acontinuous divergent wavefront. The amount of perceived divergence ofthis approximated wavefront may be varied by varying the amount ofparallax disparity between the intra-pupil images, which varies theangular range spanned by the wavefronts of light forming the intra-pupilimages. Preferably, this angular range mimics the angular range spannedby the continuous wavefront being approximated. In some embodiments, thewavefronts of light forming the intra-pupil images are collimated orquasi-collimated.

In some embodiments, the display system utilizes a light source that isconfigured to output light from a plurality of distinct light outputlocations. For example, the light source may comprise a plurality ofselectively activated light-emitting regions, with each region being adiscrete light output location. The amount of parallax disparity betweenthe intra-pupil images may be varied by changing the light outputlocations for each image. It will be appreciated that light from a givenlight output location may propagate through the display system to theeye along one path, and that light from a different light outputlocation on the light source may propagate through the display system tothe eye along a different path. Consequently, spatial differences in thelight output locations may translate into differences in the paths thatthe light takes to the eye. The different paths may correspond todifferent amounts of parallax disparity. Advantageously, in someembodiments, the amount of parallax disparity may be selected byselecting the amount of spatial displacement or separation between thelight output locations of the light source.

In some embodiments, as noted above, the light source may comprise aplurality of selectively activated light-emitting regions, each of whichcorresponds to a distinct light output location. The light-emittingregions may be disposed on a plane and form a 2D light emitter array. Insome other embodiments, the light source may comprise a linear transferlens such as a F-theta (F-θ or F-tan θ) lens, a common or shared lightemitter, and an actuator to direct the light emitted by the lightemitter along different paths through the F-theta lens. The light exitsthe light source at different locations through the F-theta lens, whichfocuses the exiting light onto an image plane. Light exiting the F-thetalens at different locations is also disposed at different locations onthe image plane, and the image plane may be considered to provide avirtual 2D light emitter array. Consequently, the individual regions ofthe light emitter array, and the locations at which light from thelinear transfer lens passes through the image plane may both be referredto herein as light output locations of the light source.

In some embodiments, the actuator may be part of a dual axisgalvanometer comprising a plurality (e.g., a pair) of mirrors that areindependently actuated on different axes to direct light from the lightemitter along the desired path of propagation. In some otherembodiments, the light source may comprise a fiber scanner and theactuator may be an actuator configured to move the fiber of the fiberscanner. The light source may also comprise or be in communication witha processing module which synchronizes the output of light by the lightsource with the location of the mirrors or fiber, and with theintra-pupil image to be displayed. For example, the mirrors or fiber maymove along a known path and the light emitter may be controlled by theprocessing module to emit light when the mirrors or fiber are at aposition corresponding to a desired light output location for aparticular intra-pupil image (and the parallax disparity associated withthat image), as discussed further herein.

The display system may also comprise a spatial light modulator betweenthe light source and projection optics for injecting light into the eye.The spatial light modulator may be configured to modulate the light fromthe light source, to encode image information in that light stream toform an intra-pupil image. Preferably, the images are injected into theeye through a projection optic that simultaneously provides an image ofthe spatial light modulator plane at or near optical infinity, or someother chosen “home plane,” and also provides an image of the lightsource at or near the viewer's pupil. Thus, both image content andprecise amounts of parallax disparity may be provided to the eye.

In some embodiments, the same spatial light modulator may be used tomodulate light to form various intra-pupil images to be provided to aneye. In some such embodiments, the active light output locations (thelight output locations from which light is actively propagating at agiven point in time) may be synchronized with the modulation by thespatial light modulator. For example, activation of a light outputlocation corresponding to one intra-pupil image may be synchronized, orsimultaneous, with the activation of display elements in the spatiallight modulator, with the display elements configured to form theintra-pupil image corresponding to a particular light-emitting region.Once another light output location corresponding to a second intra-pupilimage is activated, the appropriate, possibly different, displayelements in the spatial light modulator may be activated to form thatsecond intra-pupil image. Additional intra-pupil images may be formed bysynchronizing activation of the light output locations and the imagecontent provided by the spatial light modulator. This time-basedsequential injection of intra-pupil images to the eye may be referred toas temporal multiplexing or temporally multiplexed display of theintra-pupil images. Also, it will be appreciated that an active oractivated light output location is a location from which light isactively propagating from the light source to the spatial lightmodulator used to form the intra-pupil images.

In some other embodiments, spatial multiplexing may be utilized. In suchembodiments, different areas of the spatial light modulator (e.g.different pixels) may be dedicated to forming different intra-pupilimages. An optical mechanism may be provided between the spatial lightmodulator and the projection optic to direct light from differentregions such that the light propagates in different directions throughthe projection optic. Examples of suitable optical mechanisms includelenslet arrays. Consequently, different intra-pupil images may be formedand provided to the eye simultaneously, with the parallax disparitydetermined by the locations of the pixels forming the images and withthe optical mechanism directing the propagation of light from thosepixels. In some embodiments, a light source without selectivelyactivated light-emitting regions (e.g., a point light source) may beutilized to generate light for the display system, since the parallaxdisparity may be set using the spatial light modulator in conjunctionwith the optical mechanism.

In some other embodiments, both spatial and temporal multiplexing may beutilized. In such embodiments, the display system may include a lightsource with selectively activated light output locations, in addition tothe above-noted optical mechanism and the formation of differentintra-pupil images in different areas of the spatial light modulator.Parallax disparity may be provided using both the selective activationof light output locations and the optical mechanism in conjunction withthe simultaneous formation of different intra-pupil images in differentthe locations of the spatial light modulator.

In embodiments with temporal multiplexing, the set of intra-pupil imagesfor approximating a particular continuous wavefront are preferablyinjected into the eye too rapidly for the human visual system to detectthat the images were provided at different times. Without being limitedby theory, the visual system may perceive images formed on the retinawithin a flicker fusion threshold as being present simultaneously. Insome embodiments, approximating a continuous wavefront may includesequentially injecting beams of light for each of a set of intra-pupilimages into the eye, with the total duration for injecting all of thebeams of light being less than the flicker fusion threshold, above whichthe human visual system will perceive images as being separatelyinjected into the eye. As an example, the flicker fusion threshold maybe about 1/60 of a second. It will be appreciated that each set ofimages may consist of a particular number of parallax views, e.g., twoor more views, three or more views, four or more views, etc. and all ofthese views are provided within the flicker fusion threshold.

Preferably, the display system has a sufficiently small exit pupil thatthe depth of field provided by light forming individual intra-pupilimages is substantially infinite and the visual system operates in an“open-loop” mode in which the eye is unable to accommodate to anindividual intra-pupil image. In some embodiments, the light beamsforming individual images occupy an area having a width or diameter lessthan about 0.5 mm when incident on the eye. It will be appreciated,however, that light beams forming a set of intra-pupil images are atleast partially non-overlapping and preferably define an area largerthan 0.5 mm, to provide sufficient information to the lens of the eye toelicit a desired accommodative response based on the wavefrontapproximation formed by the wavefronts of the light forming theintra-pupil images.

Without being limited by theory, the area defined by a set of beams oflight may be considered to mimic a synthetic aperture through which aneye views a scene. It will be appreciated that viewing a scene through asufficiently small pinhole in front of the pupil provides a nearlyinfinite depth of field. Given the small aperture of the pinhole, thelens of the eye is not provided with adequate scene sampling to discerndistinct depth of focus. As the pinhole enlarges, additional informationis provided to the eye's lens, and natural optical phenomena allow alimited depth of focus to be perceived. Advantageously, the area definedby the set of beams of light and the corresponding sets ofparallactically-disparate intra-pupil images may be made larger than thepinhole producing the infinite depth of field and the multipleintra-pupil images may produce an approximation of the effect providedby the enlarged pinhole noted above.

As discussed herein, in some embodiments, the different angles at whichthe light beams propagate towards the pupil may be provided using alight source having a plurality of selectively activated light outputlocations that output light to a spatial light modulator that modulatesthe light to form the images. It will be appreciated that light fromdifferent light output locations of the light source will take differentpaths to the spatial light modulator, which in turn will take adifferent path from the spatial light modulator to the output pupil ofthe projection optic and thus to the viewer's eyes. Consequently,lateral displacement of the active light output locations translate intoangular displacement in the light leaving the spatial light modulatorand ultimately propagating towards the viewer's pupil through theprojection optic. In some embodiments, increases in lateral displacementbetween the activated light-emitting regions may be understood totranslate to increases in angular displacement as measured with respectto the spatial light modulator plane. In some embodiments, each of theintra-pupil images may be formed by outputting light from a differentlight output location, thereby providing the angular displacementbetween the beams of light forming each of the images.

In some embodiments, the light source and/or light output locations ofthe light source may change position or jitter within a single parallaximage (intra-pupil image) display episode. For example, the light sourceand/or light emitting regions may physically move and/or different lightoutput locations (e.g., the different light emitters of an array oflight emitters) may be activated to provide the desired change inposition while displaying an intra-pupil image. The speed ofdisplacement or jitter may be higher than the update rate of theparallax image on the spatial light modulator. The jittered displacementmay be in any direction, including torsional, depending on theperceptual effect that is desired.

In some embodiments, the display system may include a combiner eyepiece,which allows virtual image content to be overlaid with the viewer's viewof the world, or ambient environment. For example, the combiner eyepiecemay be an optically transmissive waveguide that allows the viewer to seethe world. In addition, the waveguide may be utilized to receive, guide,and ultimately output light forming the intra-pupil images to theviewer's eyes. Because the waveguide may be positioned between theviewer and the world, the light outputted by the waveguide may beperceived to form virtual images that are placed on depth planes in theworld. In essence, the combiner eyepiece allows the viewer to receive acombination of light from the display system and light from the world.

In some embodiments, the display system may also include an eye trackingsystem to detect the viewer's gaze direction. Such an eye trackingsystem allows appropriate content to be selected based upon where theviewer is looking.

Advantageously, by shifting the mechanism for providing divergentwavefronts from multiple, discrete light output structures, which createwavefronts with a particular associated divergence, to a singlestructure that can create an arbitrary amount of divergence, thephysical size and complexity of the system may be reduced; that is, someof the output structures may be eliminated. In addition, it may bepossible to place virtual content on a larger number of depth planesthey would be practical if each depth plane required a dedicatedstructure to create a given wavefront divergence. This increase in thenumber of depth planes may provide a more realistic and comfortableviewing experience for the viewer. In addition, in some embodiments,light from each spatial light modulator pixel may remain nominallycollimated, thereby facilitating integration of a projection systemhaving that spatial light modulator with combiner eyepieces that utilizecollimated pixel light.

Reference will now be made to the figures, in which like referencenumerals refer to like parts throughout.

As discussed herein, the perception of an image as being“three-dimensional” or “3-D” may be achieved by providing slightlydifferent presentations of the image to each eye of the viewer. FIG. 2illustrates a conventional display system for simulatingthree-dimensional imagery for a user. Two distinct images 190, 200—onefor each eye 210, 220—are outputted to the user. The images 190, 200 arespaced from the eyes 210, 220 by a distance 230 along an optical orz-axis that is parallel to the line of sight of the viewer. The images190, 200 are flat and the eyes 210, 220 may focus on the images byassuming a single accommodated state. Such 3-D display systems rely onthe human visual system to combine the images 190, 200 to provide aperception of depth and/or scale for the combined image.

It will be appreciated, however, that the human visual system is morecomplicated and providing a realistic perception of depth is morechallenging. For example, many viewers of conventional “3-D” displaysystems find such systems to be uncomfortable or may not perceive asense of depth at all. Without being limited by theory, it is believedthat viewers of an object may perceive the object as being“three-dimensional” due to a combination of vergence and accommodation.Vergence movements (i.e., rotation of the eyes so that the pupils movetoward or away from each other to converge the lines of sight of theeyes to fixate upon an object) of the two eyes relative to each otherare closely associated with focusing (or “accommodation”) of the lensesand pupils of the eyes. Under normal conditions, changing the focus ofthe lenses of the eyes, or accommodating the eyes, to change focus fromone object to another object at a different distance will automaticallycause a matching change in vergence to the same distance, under arelationship known as the “accommodation-vergence reflex,” as well aspupil dilation or constriction. Likewise, a change in vergence willtrigger a matching change in accommodation of lens shape and pupil size,under normal conditions. As noted herein, many stereoscopic or “3-D”display systems display a scene using slightly different presentations(and, so, slightly different images) to each eye such that athree-dimensional perspective is perceived by the human visual system.Such systems are uncomfortable for many viewers, however, since they,among other things, simply provide a different presentation of a scene,but with the eyes viewing all the image information at a singleaccommodated state, and work against the “accommodation-vergencereflex.” Display systems that provide a better match betweenaccommodation and vergence may form more realistic and comfortablesimulations of three-dimensional imagery, contributing to increasedduration of wear.

FIG. 4 illustrates aspects of an approach for simulatingthree-dimensional imagery using multiple depth planes. With reference toFIG. 3 , objects at various distances from eyes 210, 220 on the z-axisare accommodated by the eyes 210, 220 so that those objects are infocus; that is, the eyes 210, 220 assume particular accommodated statesto bring into focus objects at different distances along the z-axis.Consequently, a particular accommodated state may be said to beassociated with a particular one of depth planes 240, with has anassociated focal distance, such that objects or parts of objects in aparticular depth plane are in focus when the eye is in the accommodatedstate for that depth plane. In some embodiments, three-dimensionalimagery may be simulated by providing different presentations of animage for each of the eyes 210, 220, with the presentations of the imagealso being different for different depth planes. While shown as beingseparate for clarity of illustration, it will be appreciated that thefields of view of the eyes 210, 220 may overlap, for example, asdistance along the z-axis increases. In addition, while shown as flatfor ease of illustration, it will be appreciated that the contours of adepth plane may be curved in physical space, such that all features in adepth plane are in focus with the eye in a particular accommodatedstate.

The distance between an object and the eye 210 or 220 may also changethe amount of divergence of light from that object, as viewed by thateye. FIGS. 4A-4C illustrate relationships between distance and thedivergence of light rays. The distance between the object and the eye210 is represented by, in order of decreasing distance, R1, R2, and R3.As shown in FIGS. 4A-4C, the light rays become more divergent asdistance to the object decreases. As distance increases, the light raysbecome more collimated. Stated another way, it may be said that thelight field produced by a point (the object or a part of the object) hasa spherical wavefront curvature, which is a function of how far away thepoint is from the eye of the user. The curvature increases withdecreasing distance between the object and the eye 210. Consequently, atdifferent depth planes, the degree of divergence of light rays is alsodifferent, with the degree of divergence increasing with decreasingdistance between depth planes and the viewer's eye 210. While only asingle eye 210 is illustrated for clarity of illustration in FIGS. 4A-4Cand other figures herein, it will be appreciated that the discussionsregarding eye 210 may be applied to both eyes 210 and 220 of a viewer.

Without being limited by theory, it is believed that the human eyetypically can interpret a finite number of depth planes to provide depthperception. Consequently, a highly believable simulation of perceiveddepth may be achieved by providing, to the eye, different presentationsof an image corresponding to each of these limited number of depthplanes. The different presentations may be separately focused by theviewer's eyes, thereby helping to provide the user with depth cues basedon the accommodation of the eye required to bring into focus differentimage features for the scene located on different depth planes and/orbased on observing different image features on different depth planesbeing out of focus.

Because each depth plane has an associated wavefront divergence, todisplay image content appearing to be at a particular depth plane, somedisplays may utilize waveguides that have optical power to output lightwith a divergence corresponding to that depth plane. A plurality ofsimilar waveguides, but having different optical powers, may be utilizedto display image content on a plurality of depth planes. For example,such systems may utilize a plurality of such waveguides formed in astack. FIG. 5 illustrates an example of a waveguide stack for outputtingimage information to a user. A display system 250 includes a stack ofwaveguides 260 that may be utilized to provide three-dimensionalperception to the eye/brain using a plurality of waveguides 270, 280,290, 300, 310 to output image information. Image injection devices 360,370, 380, 390, 400 may be utilized to inject light containing imageinformation into the waveguides 270, 280, 290, 300, 310. Each waveguide270, 280, 290, 300, 310 may include a structure (e.g., an opticalgrating and/or lens 570, 580, 590, 600, 610, respectively) that providesoptical power, such that each waveguide outputs light with a presetamount of wavefront divergence, which corresponds to a particular depthplane. Thus, each waveguide 270, 280, 290, 300, 310 places image contenton an associated depth plane determined by the amount of wavefrontdivergence provided by that waveguide.

It will be appreciated, however, that the one-to-one correspondencebetween a waveguide and a depth plane may lead to a bulky and heavydevice in systems in which multiple depth planes are desired. In suchembodiments, multiple depth planes would require multiple waveguides. Inaddition, where color images are desired, even larger numbers ofwaveguides may be required, since each depth plane may have multiplecorresponding waveguides, one waveguide for each component color may berequired to form the color images.

Advantageously, various embodiments may provide a simpler display systemthat approximates a desired continuous wavefront by using discrete lightbeams that form intra-pupil images that present different parallax viewsof an object or scene.

With reference now to FIG. 6A, the pre-accommodation andpost-accommodation conditions of an eye 210 upon receiving a continuousinput wavefront 1000 are illustrated. Illustration a) shows thepre-accommodation condition, before the visual system brings thewavefront 1000 into focus on the retina 211. Notably, the focal point212 is not on the retina 211. For example, the focal point 212 may beforward of the retina 211 as illustrated. Illustration b) shows thepost-accommodation condition, after the human visual system flexespupillary musculature of the eye 210 of the viewer to bring thewavefront 1000 into focus on the retina 211. As illustrate, the focalpoint 212 may be on the retina 211.

It has been found that a continuous wavefront such as the wavefront 1000of FIG. 6A may be approximated using a plurality of wavefronts. FIG. 6Billustrates the pre-accommodation and post-accommodation conditions ofthe eye 210 upon receiving a piecewise approximation of the continuouswavefront 1000 of FIG. 6A. Illustration a) of FIG. 6B shows thepre-accommodation condition and illustration b) shows thepost-accommodation condition of the eye 210. The approximation may beformed using a plurality of constituent wavefronts 1010 a, 1010 b, and1010 c, each of which is associated with separate beams of light. Asused herein, references numerals 1010 a, 1010 b, and 1010 c may indicateboth a light beam and that light beam's associated wavefront. In someembodiments, the constituent wavefronts 1010 a and 1010 b may be planarwavefronts, such as formed by a collimated beam of light. As shown inillustration b), the wavefront approximation 1010 formed by theconstituent wavefronts 1010 a and 1010 b are focused by the eye 210 ontothe retina 211, with the focal point 212 on the retina 211.Advantageously, the pre- and post-accommodation conditions are similarto that caused by the continuous wavefront 1000 shown in FIG. 6A.

It will be appreciated that continuous divergent wavefronts may beformed using optical projection systems. FIG. 7A illustrates an eyeaccommodating to a divergent wavefront emanating from a finitefocal-distance virtual image provided by a projection system. The systemincludes a spatial light modulator 1018 and projection optics 1020 withfocal length “F” and an external stop. An image may be formed by thespatial light modulator 1018 and light from the spatial light modulator1018 containing the image information may be directed through projectionoptics 1020 to the eye 210. As indicated in FIG. 7A, the spacing (lessthan F) between the spatial light modulator 1018 and the projectionoptics 1020 may be chosen such that a divergent wavefront 1000 isoutputted towards the eye 210. As noted above regarding FIG. 6A, the eye210 may then focus the wavefront 1000 on the retina 211.

FIG. 7B illustrates a system for forming an approximation of thedivergent wavefront of FIG. 7A utilizing wavefront segments formed byinfinity-focused virtual images. As above, the system includes spatiallight modulator 1018 and projection optics 1020. The spatial lightmodulator 1018 forms two images that are shifted relative to oneanother. The spatial light modulator 1018 is placed at distance F fromthe back focal plane of projection optics 1020, which have a back focallength of F. Light beam 1010 a, containing image information for a firstimage, propagates through the projection optics 1020 into the eye 210.Light beam 1010 b containing image information for a second image takesa different path through the projection optics 1020 into the eye 210. Asdiscussed herein, the light beams 1010 a and 1010 b may be emitted fromdifferent regions of a light source (not illustrated), thereby causingthose light beams to illuminate the spatial light modulator 1018 fromdifferent angles, which in turn causes images formed by the light beams1010 a and 1010 b to be spatially shifted relative to one another. Thelight beams 1010 a and 1010 b propagate away from the spatial lightmodulator along paths through the projection optics 1020 and into theeye 210 such that those light beams define an angular range, from onelight beam to the other, that matches the angular range of the divergentwavefront 1000 (FIG. 7A). It will be appreciated that the angularseparation between light beams 1010 a and 1010 b increases withincreases in the amount of wavefront divergence that is approximated. Insome embodiments, the projection optics 1020 and the spacing between thespatial light modulator 1018 and the projection optics 1020 areconfigured such that each of the light beams 1010 a and 1010 b arecollimated.

With reference now to FIG. 8 , examples of parallax views forming thedivergent wavefront approximation of FIG. 7B are illustrated. It will beappreciated that each of light beams 1010 a, 1010 b, and 1010 c form adistinct image of one view of the same objects or scene from slightlydifferent perspectives corresponding to the different placements of theimages in space. As illustrated, the images may be injected into the eye210 sequentially at different times. Alternatively, the images may beinjected simultaneously if the optical system permits, or the images canbe injected in groups, as discussed herein. In some embodiments, thetotal duration over which light forming all of the images is injectedinto the eye 210 is less than the flicker fusion threshold of theviewer. For example, the flicker fusion threshold may be 1/60 of asecond, and all of the light beams 1010 a, 1010 b, and 1010 c areinjected into the eye 210 over a duration less than that flicker fusionthreshold. As such, the human visual system integrates all of theseimages and they appear to the eye 210 as if the light beams 1010 a, 1010b, and 1010 c were simultaneously injected into that eye 210. The lightbeams 1010 a, 1010 b, and 1010 c thus form the wavefront approximation1010.

With reference now to FIG. 9 , an example of a display system 1001comprising a projection system 1003 for forming the divergent wavefrontapproximation 1010 of FIG. 7B is illustrated. The projection system 1003comprises a light source 1026 configured to output light 1010 a′ and1010 b′ to a spatial light modulator 1018, which modulates the light toform images showing slightly different parallax views of the same objector scene. The modulated light with the image information then propagatesthrough the relay/projection optics 1020, and is outputted by therelay/projection optics 1020 as light beams 1010 a and 1010 b into theeye 210. The projection system 1003 may also include a lens structure1014, which may be configured to convert the spatial differences in theemission of the light 1010 a′ and 1010 b′ into angular differences inthe propagation of that light to the spatial light modulator 1018. Theprojection system 1003 may further include a polarizing beam splitter1016 configured to 1) direct light from the light source 1026 to thespatial light modulator 1018; and 2) permit modulated light from thespatial light modulator 1018 to propagate back through the beam splitter1016 to the relay/projection optics 1020. In some embodiments, thedisplay system 1001 may include an eye tracking device 1022, e.g., acamera, configured to monitor the gaze of the eye. Such monitoring maybe used to determine the direction in which the viewer is looking, whichmay be used to select image content appropriate for that direction.Preferably, the eye tracking device 1022 tracks both eyes of the viewer,or each eye includes its own associated eye tracking device. As aresult, vergence of both eyes of the viewer may be tracked and theconvergence point of the eyes may be determined to determine in whatdirection and at what distance the eyes are directed.

It will be appreciated that the light 1010 a′ and 1010 b′ may beoutputted by the light source 1026 at different times, the spatial lightmodulator 1018 may form the different parallax views with the light 1010a′ and 1010 b′ at different times, and the resultant light beams 1010 aand 1010 b may be injected into the eye 210 at different times, asdiscussed herein.

With continued reference to FIG. 9 , the light source 1026 may be a 2Dlight source having a plurality of selectively-activated light outputlocations disposed substantially on a plane. In some embodiments, theselectively-activated light output locations may be selectivelyactivated light-emitting regions. For example, the light source 1026 maybe a light-emitting diode (LED) array, or a spatial light modulator(e.g., a digital micromirror device such as a digital light processing(DLP) device, a LCOS device, etc.) containing an array of discrete unitsor light emitters that output light. Examples of LED arrays includeorganic light-emitting diode (OLED) arrays, and inorganic light-emittingdiode (ILED) arrays. In some embodiments, individual light-emittingdiodes and/or light modulators in the light source 1026 may constitute alight-emitting region. In some other embodiments, groups oflight-emitting diodes and/or light modulators may form light-emittingregions. In such embodiments, there may be some overlap between thelight-emitting diodes and/or light modulators of differentlight-emitting regions although the regions may be considered distinctbecause the overlap is not complete.

In some other embodiments, the light source 1026 may be configured tofocus light onto an image plane to, in effect, provide a virtual 2Dlight source on that image plane. Different locations on the image planemay be considered to be different light output locations and thoselocations may be activated by directing light through those locations onthe image plane using actuated mirrors or a fiber scanner to steer lightfrom a light emitter. Further details regarding such virtual 2D lightsources are provided below in the discussion of FIGS. 19 and 20 .

In some embodiments, examples of spatial light modulators 1018 includeliquid crystal on silicon (LCOS) panels. As another example, in someother embodiments, spatial light modulator 1018 may comprise atransmissive liquid crystal panel or a MEMs device, such as a DLP.

With continued reference to FIG. 9 , the display system 1001 may alsoinclude control systems 1024 for determining the timing and the type ofimage content provided by the display system. In some embodiments, thecontrol system 1024 comprises one or more hardware processors withmemory storing programs for controlling the display system 1001. Forexample, the system 1024 may be configured to control activation of thelight-emitting regions of the light source 1026, the actuation ofindividual pixel elements of the spatial light modulator 1018, and/orthe interpretation and reaction of the display system 1001 to datareceived from the eye tracking device 1022. Preferably, the system 1024includes a computation module 1024 a configured to receive an inputregarding a desired depth plane or wavefront divergence and to calculatethe appropriate light-emitting regions to activate, in order to formparallax views with the proper amount of disparity for the desired depthplane or wavefront divergence. In addition, computation module 1024 amay be configured to determine the appropriate actuation of the pixelsof the spatial light modulator 1018 to form images of the desiredparallax views. The system 1024 may also include a synchronizationmodule 1024 b that is configured to synchronize the activation ofparticular light-emitting regions of the light source 1026 withmodulation of light by the spatial light modulator 1018 to form imagesto provide the parallax view corresponding to those activatedlight-emitting regions. In addition, the system 1024 may include an eyetracking module 1024 c that receives inputs from the eye tracking device1022. For example, the eye tracking device 1022 may be a cameraconfigured to image the eye 210. Based on images captured by the eyetracking device 1022, the eye tracking module 1024 c may be configuredto determine the orientation of the pupil and to extrapolate the line ofsight of the eye 210. This information may be electronically conveyed tothe computation module 1024 a. The computation module 1024 a may beconfigured to select image content based upon the line of sight or thegaze of the eye 210 (preferably also based upon the line of sight orgaze of the other eye of the viewer).

Because the light source 1026 may include arrays of discrete lightemitters, the size and shape of the light-emitting regions formed by thelight emitters may be varied as desired by activating selected ones ofthe light emitters. FIG. 10 illustrates examples of sizes, shapes, anddistributions for the light-emitting regions. It will be appreciatedthat the light and dark areas in the figure indicate different emittingregions that are activated for different parallax views. Example a)shows elongated light-emitting regions that are horizontally spacedapart, which may be desirable for horizontal parallax-only drivenaccommodation. Example b) shows circular light-emitting regions withboth horizontal and vertical displacement. Example c) showslight-emitting regions that have a luminance fall-off. Example d) showslight-emitting regions that overlap. Example e) shows light-emittingregions that form arrays. As indicated by the illustrated examples, thelight source 1026 (FIG. 9 ) may include light emitters that are binary(which simply turn on and off) and/or light emitters that incorporategrayscale (which emit light of selectively variable intensity). In someembodiments, the light source 1026 may include elements that switch atvery high rates, including rates beyond the parallax-switching rate forthe system 1001. For example, the light source 1026 may have lightoutputting elements that switch the light output on and off at a ratehigher than the rate at which the parallax (intra-pupil) images areswitched in embodiments in which different intra-pupil images aredisplayed at different times.

With reference again to FIG. 9 , in some embodiments, the control system1024 may include two parts: 1) light field generation and 2) factoredlight field optimization. As discussed herein, to approximate awavefront, an appropriate image is displayed on the spatial lightmodulator 1018 for each activated light-emitting region of the lightsource 1026. It will be appreciated that these images are created duringthe light field generation step, where a 3D scene is rendered frommultiple, slightly offset viewpoints corresponding to the slight shiftsin activated light-emitting regions. For example, to display a 5×5 lightfield, the 3D scene would be rendered 25 times from 25 differentviewpoints that are arranged in a grid pattern. The location of theviewpoint in the grid pattern would correspond to the location of theactivated light source region, and the rendered image would correspondto the image formed by the spatial light modulator.

It may be desirable to increase the brightness of images formed by thespatial light modulator 1018. Advantageously, utilizing a light source1026 comprising an array of light emitters allows the formation oflight-emitting regions having a variety of shapes and sizes, which maybe utilized to increase brightness. In some embodiments, brightness maybe increased by increasing the size of the activated light-emittingregion without significantly changing the image formed by the spatiallight modulator 1018. The computation module 1024 a may be configured todetermine the size and shape of the activated light-emitting regionusing factored light field optimization. The module 1024 a may beconfigured to take an input focal stack and create a series of patternsto be displayed on the spatial light modulator 1018 as well as on thelight source 1026, with the patterns configured create a desiredapproximation to the focal stack in the least squared sense. Theoptimization takes advantage of the fact that small shifts in theviewpoint do not significantly change the perceived image, and is ableto generate light-emitting region patterns utilizing illumination from alarger area on the light source 1026, while displaying the same image onthe spatial light modulator 1018.

The optimization problem may be formulated as a non-convex optimizationproblem, given below:

$\begin{matrix}\underset{\{{A,B}\}}{\arg\min} & {\frac{1}{2}{{y - \left\{ {AB}^{T} \right\}}}_{2}^{2}} \\{{subject}{to}} & {{0 \leq A},{B \leq 1},}\end{matrix}$where the projection operator p performs the linear transformation fromthe 4D light field to the 3D focal stack (using the shift and addalgorithm). This problem is a nonnegative matrix factorization embeddedin a deconvolution problem. The algorithm solving this problem uses thealternating direction method of multipliers (ADMM). Additional detailsregarding an example method of solving this problem are discussed inAppendix I. It will be appreciated that the module 1024 a is configuredto actively calculate, in real time, the appropriate size and shape ofthe light-emitting region based upon the parallax view to be formed by aspatial light modulator 1018.

In some other embodiments, the optimization problem may be formulated asa slightly different non-convex optimization problem, as given below:

$\begin{matrix}\underset{\{{A,B}\}}{\arg\min} & {\frac{1}{2}{{y - \left\{ {AB}^{T} \right\}}}_{2}^{2}} \\{{subject}{to}} & {{0 \leq A},{B \leq 1},}\end{matrix}$where A and B represent the patterns displayed on the spatial lightmodulators (e.g., the light source 1026 and the spatial light modulator1018 for forming images), y is the target 4D light field that is thedesired output of the algorithm, and AB′ is the operator combining thespatial light modulator patterns to simulate the 4D light field emittedby the physical display when A and B are shown on the modulators. Thisproblem is a nonnegative matrix factorization. The algorithm solvingthis problem uses an iterative optimization technique to refine A and Bfrom a random initial guess.

With continued reference to FIG. 9 , it will be appreciated that theflicker fusion threshold of the human visual system places a timeconstraint on the number of images that may be injected into the eye 210while still being perceived as being injected simultaneously. Forexample, the processing bandwidth of the control system 1024 and theability to switch light-emitting regions of the light source 1026 andlight modulators of the spatial light modulator 1018 may limit thenumber of images that may be injected into the eye 210 within theduration allowed by the flicker fusion threshold. Given this finitenumber of images, the control system 1024 may be configured to makechoices regarding the images that are displayed. For example, within theflicker fusion threshold, the display system may be required to inject aset of parallactically-disparate intra-pupil images into the eye, and inturn each parallax view may require images of various component colorsin order to form a full color image. In some embodiments, the formationof full color images using component color images is bifurcated from theelucidation of a desired accommodation response. For example, withoutbeing limited by theory, it may be possible to elicit the desiredaccommodation response with a single color of light. In such a case, theparallactically-disparate intra-pupil images used to elicit theaccommodation response would only be in the single color. As a result,it would not be necessary to form parallactically-disparate intra-pupilimages using other colors of light, thereby freeing up time within theflicker fusion threshold for other types of images to be displayed. Forexample, to better approximate the wavefront, a larger set ofparallactically-disparate intra-pupil images may be generated.

In some other embodiments, the control system 1024 may be configured todevote less time within the flicker fusion threshold for displayingimages of colors of light for which the human visual system is lesssensitive. For example, the human visual system is less sensitive toblue light then green light. As a result, the display system may beconfigured to generate a higher number of images formed with green lightthan images formed with blue light.

With reference now to FIG. 11A, another example of a projection system1003 for forming the divergent wavefront approximation of FIG. 7B isillustrated. Preferably, the projection system produces a relativelylong depth of field, which may be controlled by the limiting aperture inthe system. Without being limited by theory, projection systemsproviding images to the eye with an effective pupil diameter ofapproximately 0.5 mm are believed to force the human visual system tooperate in “open-loop” mode, as the eye is unable to accommodate to suchimages. By providing images with such an effective pupil diameter, thedisplay system reduces the spot-size on the retina for aninfinity-focused image.

With continued reference to FIG. 11A, the projection system 1003 formsimages with parallax disparity as discussed herein. The images may berapidly, alternatingly provided to the eye of a viewer at a rate that ishigher than the perception persistence of the human visual system(e.g. >60 Hz). As discussed herein, the illustrated projection system1003 simultaneously produces an image of the illumination source at afinite conjugate plane, and an image of the pixel (image) source atinfinity. In addition, selectively activated light-emitting regions 1026a and 1026 b are spaced to produce displacement of the optical systempupil to align the parallax images with respect to each other within theviewer pupil.

With reference now to FIG. 11B, an example of a range of depth planesprovided by the projection system of FIG. 11A is illustrated. The rangespans from a far plane at optical infinity to a near plane closer to theeye 210. The far plane at optical infinity may be provided by collimatedlight beams 1010 a and 1010 b. The near plane may be provided usingspatially displaced activated light-emitting regions, as disclosedherein, and may be understood to be the nearest depth plane provided bythe display system. In some embodiments, the perceived nearness of thenear plane to the eye 210 may be determined by the maximum parallaxdisparity between the light beams 1010 a and 1010 b, which may bedetermined by the maximum distance that the display system allows theselectively activated light-emitting regions 1026 a and 1026 b to beseparated while still forming a clear image of the light source 1026 ator near the viewer's pupil.

Advantageously, as discussed herein, the use of a light source 1026comprising a plurality of discrete, selectively activated light emittersprovides the ability to produce a broad range of pupil or perceivedimage shapes, luminance profiles, and arrays to achieve various depth offield effects (through manipulation of illumination source size, shapeand position). The light source 1026 advantageously also provides theability to flexibly and interactively change the shape of the pupil toaccommodate horizontal parallax only, full parallax, or othercombinations of parallax as is desired for driving accommodation whileproviding high luminous efficiency.

With reference now to FIG. 12 , an example of a light sourceconfiguration for projection systems 1003 is illustrated. The lightsource 1026 includes a single fixed illuminator 1028 and a spatial lightmodulator 1032 for regulating the output of light from the illuminator1028 to the spatial light modulator 1018 for forming images. The lightsource 1026 may also include a condenser/collimator lens to direct lightfrom the illuminator 1028 to the spatial light modulator 1032. Thespatial light modulator 1032 may include pixels and/or shutters thatallow or block light from passing through as desired. It will beappreciated that the pixels and/or shutters may be actuated to permitlight to pass and that the areas in which light passes are considered tobe light-emitting regions (e.g. light-emitting regions 1026 a and 1026b).

With reference now to FIG. 13A, an example of a projection system 1003for placing virtual objects on a default depth plane that is less thanoptical infinity is illustrated. As illustrated, the projection optics1020 may have a focal length “F” and the spatial light modulator may bepositioned at less than F, which biases the system 1003 to have a fardepth plane at less than optical infinity by causing the light beams1010 a and 1010 b to diverge. The amount that the light beams 1010 a and1010 b diverge may be determined by the position of the spatial lightmodulator 1018 relative to the projection optics 1020, with closerspacing causing greater amounts of divergence. Because a certain amountof divergence is expected as a default due to the spacing, in someembodiments, the size of the light-emitting regions 1026 a and 1026 bfor each intra-pupil image may be scaled up (e.g. by increasing thenumber of LEDs, increasing the number of light source illuminationpixels, etc. that are activated to illuminate the spatial lightmodulator when forming an intra-pupil image) and the exit pupilassociated with each intra-pupil image may be larger than 0.5 mm. As aresult, the visual system may not function in an open loop mode. In someembodiments, the size of the light-emitting regions 1026 a and 1026 bmay be set by the control system 1024 (FIG. 9 ), which may be programmedto vary the size of the light-emitting regions 1026 a and 1026 b basedupon a desired default depth plane for the system 1003. In addition, thecross-sectional widths of the light beams for forming each intra-pupilimage is preferably sufficiently large relative to the opticalstructures of the projection optics 1020 for the projection optics 1020to act on the light so as to provide the desired divergence.

With reference now to FIG. 13B, an example of a range of depth planesprovided by the projection system of FIG. 13A is illustrated. Theillustrated range spans from a far plane, at a distance of D that isless than optical infinity, to a near plane relative to the eye 210. Thefar plane may be set by appropriate selection of the position of thespatial light modulator relative to the projection optics 1020. The nearplane may be provided as disclosed above regarding FIG. 11B.

With reference now to FIG. 14 , an example of a projection system 1003configured for the spatially multiplexed display of intra-pupil imagesis illustrated. Rather than relying on spatial displacement betweenlight emitting regions to provide the desired parallax disparity,parallax disparity may be provided by utilizing different areas of thespatial light modulator 1018 to form different intra-pupil images. Anoptical mechanism 1019 is configured to direct the light from each ofthese different areas at different angles towards the projection optics1020, which outputs light beams 1010 a and 1010 b towards an eye of theviewer (not shown). In some embodiments, the areas of the spatial lightmodulator 1018 for forming different intra-pupil images may beinterleaved. For example, pixels providing image information fordifferent intra-pupil images may be interleaved with one another. Theoptical mechanism 1019 may be configured to translate the differentlocations (e.g., different pixels) from which the optical mechanism 1019receives light into the different angles at which the light from thepixels enter the projection optics 1020. In some embodiments, theoptical mechanism 1019 may include a prism and/or a lens structure suchas a lenslet array.

It will be appreciated different, non-overlapping regions of the spatiallight modulator 1018 may be dedicated to providing image information fordifferent intra-pupil images. Because these regions are distinct fromone another, they may be actuated simultaneously in some embodiments. Asa result, multiple intra-pupil images may be presented to the eyesimultaneously. This may advantageously reduce the requirements for thespeed at which the spatial light modulator 1018 is required to refreshimages. As discussed above, in order to provide the perception that allimages of a set of intra-pupil images for approximating a continuouswavefront are present simultaneously, all of these images must bepresented within the flicker fusion threshold. In some embodiments, allor a plurality of images of a set of intra-pupil images are presented atthe same time, in different regions of the spatial light modulator, suchthat rapid sequential displaying of these simultaneously presentedimages is not required for the human visual system to perceive theimages as being present simultaneously.

As illustrated, light source 1028 provides light through lens structure1014 to illuminate the spatial light modulator 1018. In someembodiments, the light source 1028 may be a single fixed illuminatorwithout any selectively activated light-emitting regions.

In some other embodiments, the light source may include selectivelyactivated light-emitting regions, which may advantageously provideadditional control over parallax disparity. Thus, the projection systemmay utilize both spatial and temporal multiplexing. With reference nowto FIG. 15 , an example of a projection system 1003 configured forspatial and temporal multiplexing of intra-pupil images is illustrated.The projection system 1003 may include the light source 1026, which mayinclude selectively activated light-emitting regions, e.g., regions 1026a and 1026 b. As discussed herein, spatial displacement between thelight emitting regions may be utilized to provide parallax disparity inthe alternately outputted light beams 1010 a, 1010 b, 1010 c, and 1010d. In addition, as discussed above regarding FIG. 14 , the projectionsystem 1000 may include the optical mechanism 1019 between the spatiallight modulator 1018 and the projection optics 1020. The spatial lightmodulator 1018 and optical mechanism 1019 may work together to providespatial multiplexing. Thus, illumination of the spatial light modulator1018 by a single light-emitting region may produce multiple intra-pupilimages. For example, activation of the light-emitting region 1026 ailluminates the spatial light modulator 1018, which simultaneouslygenerates image information for two intra-pupil images, with the lightbeams for each image directed in different directions by the opticalmechanism 1019. The light enters the projection optics 1020 and exits aslight beams 1010 b and 1010 d for forming two distinct intra-pupilimages. Similarly, subsequent activation of the light-emitting region1026 b results in light beams 1010 a and 1010 d for forming two otherintra-pupil images.

With reference to both FIGS. 14 and 15 , in some embodiments, asdiscussed above regarding FIGS. 11A-11B and 13A-13B, the locations ofthe spatial light modulator 1018 and optical mechanism 1019 relative tothe projection optics 1020 may be selected to provide a desired default“home” depth plane, which may be less than optical infinity.

The projection system 1003 shown in the various figures herein may bepart of a hybrid system utilizing a single finite focal-length eyepiece,which places objects on a non-infinity depth plane as a default, whileemploying parallax-driven accommodation to place virtual objects onother depth planes. For example, the projection system 1003 may beconfigured to have a default depth plane at 0.3 dpt or 0.5 dpt, whichmay be sufficiently close the optical infinity to fall within atolerance of the human visual system for accommodation-vergencemismatches. For example, without being limited by theory, it is believedthat the human visual system may comfortably tolerate displaying contentfrom optical infinity on a depth plane of 0.3 dpt. In such systems,beams of light 1010 a and 1010 b will have an amount of wavefrontdivergence corresponding to the default depth plane. Advantageously,such configurations may reduce the computational a load on processors(e.g., graphics processing units) in the display system, which mayprovide advantages for lowering power consumption, lowering latency,increasing processor options, among other benefits.

With reference now to FIG. 16 , an example is illustrated of aprojection system 1003 comprising a pupil relay combiner eyepiece 1030for superimposing image content on a user's view of the world.Preferably, the eyepiece 1030 is optically transmissive, allowing lightfrom the world to propagate through the eyepiece into the eye 210 of theviewer. In some embodiments, the eyepiece 1030 comprises one or morewaveguides having incoupling optical elements 770 and outcouplingoptical elements 800. The incoupling optical element 770 receive lightfrom the projection optics 1020 and redirect that light such that itpropagates through the eyepiece 1030 by total internal reflection to theout coupling optical element 800. The outcoupling optical element 800outputs the light to the viewer's eye 210. Advantageously, eyepiece 1030preserves all of the image attributes of provided by the projectionsystem 1003, and thus rapidly switching parallax views are accuratelyportrayed through the eyepiece 1030.

The incoupling optical element 770 and the outcoupling optical element800 may be refractive or reflective structures. Preferably, theincoupling optical element 770 and the outcoupling optical element 800are diffractive optical elements. Examples of diffractive opticalelements include surface relief features, volume-phase features,meta-materials, or liquid-crystal polarization gratings.

It will be appreciated that the outcoupling optical elements 800 orother optical elements forming part of the eyepiece 1030 may beconfigured to have optical power. In some embodiments, the optical powermay be chosen to correct for refractive errors of the eye 210, includingrefractive errors such as myopia, hyperopia, presbyopia, andastigmatism.

With reference now to FIG. 17 , an example is illustrated of aprojection system 1003 comprising the eye tracking system 1022 and acombiner eyepiece 1030 with a pupil expander 1034. The pupil expanderreplicates the projection system pupil across the eyepiece 1030. Sincethe pupil expander 1034 replicates the projection system pupil across alarge area that may be traversed by the viewer's pupil through eyemotion, the images formed by the spatial light modulator 1018 andlocations of the light-emitting regions of the light source 1026 can beupdated based on input from the eye tracking system 1022 in real time.Advantageously, this configuration enables a larger eyebox for morecomfortable viewing, relaxing restrictions on eye-to-combiner relativepositioning and variations in inter-pupillary distance.

With reference now to FIG. 18 , an example is illustrated of aprojection system 1003 comprising eye tracking system 1022 and acombiner eyepiece 1030 with a pupil expander 1035 configured to producea non-infinity depth plane. In some embodiments, the non-infinity depthplane may be at 3 meters, which offers an in-budget accommodation of˜2.5 meters to infinity. For example, given the tolerance of the humanvisual system for accommodation-vergence mismatches, virtual content atdistances of ˜2.5 meters to infinity from the viewer may be placed onthe 3 meter depth plane with little discomfort. In such a system, theparallactically-disparate intra-pupil images may be used to driveaccommodation for a narrower range of depth planes, possibly all closerto the viewer than the fixed “default” focal plane. In some embodiments,this system may also incorporate the eye tracking system 1022 todetermine the distance of the viewer's fixation based, e.g., on vergenceangles of both eyes of the viewer.

In some embodiments, the light source 1026 may be replaced with avirtual light source formed on the image plane of a light projectionsystem. The light projection system may include an actuator capable ofcausing a beam of light to scan across an area on the image planecorresponding to the virtual light source. To mimic the ability toactivate the discrete light-emitting areas of the light source 1026, theoutput of light by the projection system is synchronized with themovement of the actuator to cause light to be outputted to desiredlocations on the image plane at particular times. Preferably, the rateat which the actuator is able to scan the beam of light across the imageplane is sufficiently high that all desired light output locations onimage plane may be accessed during the timeframe in which any givenintra-pupil image is displayed. For example, during the amount of timethat a particular intra-pupil image is displayed, the actuator ispreferable be able to scan a beam of light at least once, and preferablya plurality of times, across the area of the image plane correspondingto the virtual 2D light source.

FIG. 19 illustrates a light source 2026 comprising mirrors for directingthe propagation of light to different light output locations. The lightsource 2026 comprises a light emitter 2028 and mirrors 2030 and 2032,which are moved by actuators 2031 and 2033, respectively. Examples oflight emitters 2028 include LED's and lasers. In some embodiments, afiber optic cable may transmit light from a remotely situated lightemitter. As illustrated, light 1010 a′, 1010 b′ propagates from thelight emitter 2028 to the mirror 2032 which reflects the light to themirror 2030 which then reflects the light to propagate through the lens2034 to focus on the intermediate image plane 1026′. The mirrors 2030and 2032 may be part of a dual-axis galvanometer, with the actuators2031 and 2033 rotating the mirrors along different axes, e.g.,orthogonal axes, thereby allowing light to be directed to an areadefined along the two axes of the image plane 1026′. In someembodiments, the actuators 2031, 2033 may be motors. The lens 2034 maybe a linear transfer lens such as a F-theta (F-θ or F-tan θ) lens andmay be configured to focus light onto the flat image plane 1026′. Insome embodiments, light rays 1010 a′, 1010 b′ propagate away from theimage plane 1026′ in a similar manner as light would propagate from thelight source 1026 (see, e.g., FIG. 9 ). In some embodiments, the lightsource 2026 may also include a collimating lens 2036 to collimate lightemitted by the light emitter 2028 before the light reaches the mirror2032.

The light source 2026 preferably also includes or is in communicationwith a processing module 2038 that controls and synchronizes the outputof light from the light emitter 2028 with the movements of the actuators2031, 2033 and the intra-pupil image to be formed. For example, theprocessing module 2038 may coordinate the movements of the mirrors 2032,2030 with the emission of light from the light emitter 2028. In someembodiments, the mirrors 2032, 2030 are continuously rotated or swiveledback and forth by the actuators 2031, 2033 on the axis on which themirror is designed to move. The emission of light (e.g., a pulse oflight) by the light emitter 2028 is timed with this movement such thatthe light is directed to a desired location on the intermediate imageplane 1026′ at a given moment in time, and this location and time arealso determined based on the intra-pupil image to be displayed (e.g.,the activation of a particular light output location coincides in timewith the display of an intra-pupil image having parallax disparityassociated with that particular light output location). In someembodiments, the emission of light from the light emitter 2028 iscontrolled by switching the light emitter 2028 between on and off states(e.g., by supplying or not supplying power, respectively, to the lightemitter). In some other embodiments, the emission of light from a lightemitter 2028 may be controlled mechanically, using a physical switchthat selectively allows or blocks light from reaching the image plane1026′.

With reference now to FIG. 20 , light source 2026 may include a fiberscanner 2027. The fiber scanner 2027 may include light emitter 2028 andan actuator 2040 which causes the fiber 2042 to move. Light 1010 a′,1010 b′ propagates out of the end of the fiber 2042 through the lens2034 and focuses on the image plane 2026′. It will be appreciated thatthe actuator 2040 may cause the fiber 2042 to move along a predefinedpath (e.g., a circular path) at a known speed. Consequently, theprocessing module 2038 may be configured to synchronize the propagationof light out of the end of the fiber 2042 with the movement of the fiber2042 such that light propagates out of the fiber 2042 at a desired lightoutput location, which is in turn synchronized with the intra-pupilimage to be displayed.

As noted above, the light source 2026 may replace the light source 1026in any of the display systems discussed. For example, the light source2026 may substitute for the light source 1026 in the projection system1003 or display system 1001 of any of FIGS. 9, 11A, 12, 13A, and 15-18 .

With reference now to FIG. 21 , an example is illustrated of an eyepiece660 (which may correspond to the eyepiece 1030, FIGS. 14-16 ) comprisinga stacked waveguide assembly for outputting light of differentwavelengths corresponding to different component colors. In someembodiments, the waveguide assembly includes waveguides 670, 680, and690. Each waveguide includes an associated in-coupling optical element(which may also be referred to as a light input area on the waveguide),with, e.g., in-coupling optical element 700 disposed on a major surface(e.g., an upper major surface) of waveguide 670, in-coupling opticalelement 710 disposed on a major surface (e.g., an upper major surface)of waveguide 680, and in-coupling optical element 720 disposed on amajor surface (e.g., an upper major surface) of waveguide 690. In someembodiments, one or more of the in-coupling optical elements 700, 710,720 may be disposed on the bottom major surface of the respectivewaveguide 670, 680, 690 (particularly where the one or more in-couplingoptical elements are reflective, deflecting optical elements). Asillustrated, the in-coupling optical elements 700, 710, 720 may bedisposed on the upper major surface of their respective waveguide 670,680, 690 (or the top of the next lower waveguide), particularly wherethose in-coupling optical elements are transmissive, deflecting opticalelements. In some embodiments, the in-coupling optical elements 700,710, 720 may be disposed in the body of the respective waveguide 670,680, 690. In some embodiments, as discussed herein, the in-couplingoptical elements 700, 710, 720 are wavelength selective, such that theyselectively redirect one or more wavelengths of light, whiletransmitting other wavelengths of light. While illustrated on one sideor corner of their respective waveguide 670, 680, 690, it will beappreciated that the in-coupling optical elements 700, 710, 720 may bedisposed in other areas of their respective waveguide 670, 680, 690 insome embodiments.

As illustrated, the in-coupling optical elements 700, 710, 720 may belaterally offset from one another. In some embodiments, each in-couplingoptical element may be offset such that it receives light without thatlight passing through another in-coupling optical element. For example,each in-coupling optical element 700, 710, 720 may be configured toreceive light from a different image injection device 360, 370, 380,390, and 400 as shown in FIG. 6 , and may be separated (e.g., laterallyspaced apart) from other in-coupling optical elements 700, 710, 720 suchthat it substantially does not receive light from the other ones of thein-coupling optical elements 700, 710, 720. In some embodiments, thein-coupling optical elements 700, 710, 720 are vertically aligned andare not laterally offset.

Each waveguide also includes associated light distributing elements,with, e.g., light distributing elements 730 disposed on a major surface(e.g., a top major surface) of waveguide 670, light distributingelements 740 disposed on a major surface (e.g., a top major surface) ofwaveguide 680, and light distributing elements 750 disposed on a majorsurface (e.g., a top major surface) of waveguide 690. In some otherembodiments, the light distributing elements 730, 740, 750, may bedisposed on a bottom major surface of associated waveguides 670, 680,690, respectively. In some other embodiments, the light distributingelements 730, 740, 750, may be disposed on both top and bottom majorsurface of associated waveguides 670, 680, 690, respectively; or thelight distributing elements 730, 740, 750, may be disposed on differentones of the top and bottom major surfaces in different associatedwaveguides 670, 680, 690, respectively.

The waveguides 670, 680, 690 may be spaced apart and separated by, e.g.,gas, liquid, and/or solid layers of material. For example, asillustrated, layer 760 a may separate waveguides 670 and 680; and layer760 b may separate waveguides 680 and 690. In some embodiments, thelayers 760 a and 760 b are formed of low refractive index materials(that is, materials having a lower refractive index than the materialforming the immediately adjacent one of waveguides 670, 680, 690).Preferably, the refractive index of the material forming the layers 760a, 760 b is 0.05 or more, or 0.10 or less than the refractive index ofthe material forming the waveguides 670, 680, 690. Advantageously, thelower refractive index layers 760 a, 760 b may function as claddinglayers that facilitate TIR of light through the waveguides 670, 680, 690(e.g., TIR between the top and bottom major surfaces of each waveguide).In some embodiments, the layers 760 a, 760 b are formed of air. Whilenot illustrated, it will be appreciated that the top and bottom of theillustrated set 660 of waveguides may include immediately neighboringcladding layers.

With continued reference to FIG. 21 , light rays 770, 780, 790 areincident on and injected into the waveguides 670, 680, 690 by projectionsystem 1003 (FIGS. 9 and 11-16 ).

In some embodiments, the light rays 770, 780, 790 have differentproperties, e.g., different wavelengths or different ranges ofwavelengths, which may correspond to different colors. The in-couplingoptical elements 700, 710, 720 each deflect the incident light such thatthe light propagates through a respective one of the waveguides 670,680, 690 by TIR.

For example, in-coupling optical element 700 may be configured todeflect ray 770, which has a first wavelength or range of wavelengths.Similarly, the transmitted ray 780 impinges on and is deflected by thein-coupling optical element 710, which is configured to deflect light ofa second wavelength or range of wavelengths. Likewise, the ray 790 isdeflected by the in-coupling optical element 720, which is configured toselectively deflect light of third wavelength or range of wavelengths.

With continued reference to FIG. 21 , the in-coupled light rays 770,780, 790 are deflected by the in-coupling optical elements 700, 710,720, respectively, and then propagate by TIR within the waveguides 670,680, 690, respectively. The light rays 770, 780, 790 then impinge on thelight distributing elements 730, 740, 750, respectively. The lightdistributing elements 730, 740, 750 deflect the light rays 770, 780, 790so that they propagate towards the out-coupling optical elements 800,810, 820, respectively.

In some embodiments, the light distributing elements 730, 740, 750 areorthogonal pupil expanders (OPE's). In some embodiments, the OPE's bothdeflect or distribute light to the out-coupling optical elements 800,810, 820 and also increase the beam or spot size of this light as itpropagates to the out-coupling optical elements. In some embodiments,e.g., where the beam size is already of a desired size, the lightdistributing elements 730, 740, 750 may be omitted and the in-couplingoptical elements 700, 710, 720 may be configured to deflect lightdirectly to the out-coupling optical elements 800, 810, 820. In someembodiments, the out-coupling optical elements 800, 810, 820 are exitpupils (EP's) or exit pupil expanders (EPE's) that direct light in aviewer's eye 210 (FIGS. 15-16 ). It will be appreciated that the OPE'smay be configured to increase the dimensions of the eye box in at leastone axis and the EPE's may be to increase the eye box in an axiscrossing, e.g., orthogonal to, the axis of the OPEs.

Accordingly, in some embodiments, the eyepiece 660 includes waveguides670, 680, 690; in-coupling optical elements 700, 710, 720; lightdistributing elements (e.g., OPE's) 730, 740, 750; and out-couplingoptical elements (e.g., EP's) 800, 810, 820 for each component color.The waveguides 670, 680, 690 may be stacked with an air gap/claddinglayer between each one. The in-coupling optical elements 700, 710, 720redirect or deflect incident light (with different in-coupling opticalelements receiving light of different wavelengths) into its waveguide.The light then propagates at an angle which will result in TIR withinthe respective waveguide 670, 680, 690. In the example shown, light ray770 (e.g., blue light) is deflected by the first in-coupling opticalelement 700, and then continues to bounce down the waveguide,interacting with the light distributing element (e.g., OPE's) 730 andthen the out-coupling optical element (e.g., EPs) 800, in a mannerdescribed earlier. The light rays 780 and 790 (e.g., green and redlight, respectively) will pass through the waveguide 670, with light ray780 impinging on and being deflected by in-coupling optical element 710.The light ray 780 then bounces down the waveguide 680 via TIR,proceeding on to its light distributing element (e.g., OPEs) 740 andthen the out-coupling optical element (e.g., EP's) 810. Finally, lightray 790 (e.g., red light) passes through the waveguide 690 to impinge onthe light in-coupling optical elements 720 of the waveguide 690. Thelight in-coupling optical elements 720 deflect the light ray 790 suchthat the light ray propagates to light distributing element (e.g., OPEs)750 by TIR, and then to the out-coupling optical element (e.g., EPs) 820by TIR. The out-coupling optical element 820 then finally out-couplesthe light ray 790 to the viewer, who also receives the out-coupled lightfrom the other waveguides 670, 680.

With reference now to FIG. 22 , an example is illustrated of a wearabledisplay system 60. The display system 60 may correspond to the displaysystem 1001 of FIG. 9 , with a projection system 1003 for each eye ofthe viewer or user 90.

The display system 60 includes a display 70, and various mechanical andelectronic modules and systems to support the functioning of thatdisplay 70. The display 70 may be coupled to a frame 80, which iswearable by a display system user or viewer 90 and which is configuredto position the display 70 in front of the eyes of the user 90. Thedisplay 70 may be considered eyewear in some embodiments. In someembodiments, a speaker 100 is coupled to the frame 80 and configured tobe positioned adjacent the ear canal of the user 90 (in someembodiments, another speaker, not shown, is positioned adjacent theother ear canal of the user to provide stereo/shapeable sound control).In some embodiments, the display system may also include one or moremicrophones 110 or other devices to detect sound. In some embodiments,the microphone is configured to allow the user to provide inputs orcommands to the system 60 (e.g., the selection of voice menu commands,natural language questions, etc.), and/or may allow audio communicationwith other persons (e.g., with other users of similar display systems.The microphone may further be configured as a peripheral sensor tocollect audio data (e.g., sounds from the user and/or environment). Insome embodiments, the display system may also include a peripheralsensor 120 a, which may be separate from the frame 80 and attached tothe body of the user 90 (e.g., on the head, torso, an extremity, etc. ofthe user 90). The peripheral sensor 120 a may be configured to acquiredata characterizing the physiological state of the user 90 in someembodiments. For example, the sensor 120 a may be an electrode.

With continued reference to FIG. 22 , the display 70 is operativelycoupled by communications link 130, such as by a wired lead or wirelessconnectivity, to a local data processing module 140 which may be mountedin a variety of configurations, such as fixedly attached to the frame80, fixedly attached to a helmet or hat worn by the user, embedded inheadphones, or otherwise removably attached to the user 90 (e.g., in abackpack-style configuration, in a belt-coupling style configuration).Similarly, the sensor 120 a may be operatively coupled by communicationslink 120 b, e.g., a wired lead or wireless connectivity, to the localprocessor and data module 140. The local processing and data module 140may comprise a hardware processor, as well as digital memory, such asnon-volatile memory (e.g., flash memory or hard disk drives), both ofwhich may be utilized to assist in the processing, caching, and storageof data. The data may include data a) captured from sensors (which maybe, e.g., operatively coupled to the frame 80 or otherwise attached tothe user 90), such as image capture devices (such as cameras),microphones, inertial measurement units, accelerometers, compasses, GPSunits, radio devices, gyros, and/or other sensors disclosed herein;and/or b) acquired and/or processed using remote processing module 150and/or remote data repository 160 (including data relating to virtualcontent), possibly for passage to the display 70 after such processingor retrieval. The local processing and data module 140 may beoperatively coupled by communication links 170, 180, such as via a wiredor wireless communication links, to the remote processing module 150 andremote data repository 160 such that these remote modules 150, 160 areoperatively coupled to each other and available as resources to thelocal processing and data module 140. In some embodiments, the localprocessing and data module 140 may include one or more of the imagecapture devices, microphones, inertial measurement units,accelerometers, compasses, GPS units, radio devices, and/or gyros. Insome other embodiments, one or more of these sensors may be attached tothe frame 80, or may be standalone structures that communicate with thelocal processing and data module 140 by wired or wireless communicationpathways. In some embodiments, the local processing and data module 140may include one or more graphics processors, and may correspond to thecontrol system 1024 (FIG. 9 ).

With continued reference to FIG. 22 , in some embodiments, the remoteprocessing module 150 may comprise one or more processors configured toanalyze and process data and/or image information. In some embodiments,the remote data repository 160 may comprise a digital data storagefacility, which may be available through the internet or othernetworking configuration in a “cloud” resource configuration. In someembodiments, the remote data repository 160 may include one or moreremote servers, which provide information, e.g., information forgenerating augmented reality content, to the local processing and datamodule 140 and/or the remote processing module 150. In some embodiments,all data is stored and all computations are performed in the localprocessing and data module, allowing fully autonomous use from a remotemodule.

Examples—Light Field and Focal Stack Factorization

Light field and focal stack factorization may be utilized to determinethe light output of the display system 1001, including the outputs ofthe light source 1026, 2026 and the spatial light modulator 1018.Details regarding the factorization are discussed below.

1. Focal Stack Factorization

A focal stack y is factored into a series of time-multiplexed patternsto be displayed on two spatial light modulators A and B, which arelocated in the pupil and image plane, respectively. In some embodiments,spatial light modulators A and B may correspond to the light source1026, 2026 and the spatial light modulator 1018, respectively. Allquantities are vectorized, such that the focal stack y∈

₊ ^(m×n×s), which has a vertical resolution of m pixels, a horizontalresolution of n pixels, and s focal slices, will be represented as asingle vector y∈

₊ ^(mns). Bold-face symbols are used below for discrete vectors. Unlessotherwise specified, different color channels are ignored and assumed tobe independent. Table 1 provides an overview of tensor notification andoperators employed herein.

TABLE 1 Overview of tensor notation and operators. NotationInterpretation α scalar a vector A matrix a∘b vector outer product A 

 B Hadamard matrix product (elementwise product) AØ 

 B Hadamard matrix division (elementwise division)

The spatial light modulator in the image plane B also has a resolutionof m×n pixels, but in addition k time-multiplexed patterns may be shownin quick succession such that they will be perceptually averaged by theviewer. These spatio-temporal patterns are represented as the matrix B∈

₊ ^(mn×k) such that all spatial pixels are vectorized and form the rowindices of this matrix and the k time steps are the column indices ofthe matrix. Similarly, the pupil-plane SLM A will be represented as thematrix A∈

₊ ^(o×k), where o is the total number of addressable SLM pixels in thepupil plane and the column indices again are the time steps.

Accordingly, the goal of factoring a focal stack y into a set oftime-multiplexed patterns may be written as the non-convex optimizationproblem

$\begin{matrix}\begin{matrix}\underset{\{{A,B}\}}{\arg\min} & {\frac{1}{2}{{y - \left\{ {AB}^{T} \right\}}}_{2}^{2}} \\{{subject}{to}} & {{0 \leq A},{B \leq 1},}\end{matrix} & (1)\end{matrix}$where the projection operator

:

^(o×mn)→

^(mns) performs the linear transformation from the 4D light field to the3D focal stack (using the shift+add algorithm). This problem is anonnegative matrix factorization embedded in a deconvolution problem.The alternating direction methods of multipliers (ADMM) (as described byBoyd et al, 2001, “Distributed optimization and statistical learning viathe alternating direction method of multipliers”, Foundations and Trendsin Machine Learning 3, 1, 1-122) may be used to solve it.

To bring Equation 1 into the standard ADMIVI form, it may be rewrittenas the equivalent problem

$\begin{matrix}{\begin{matrix}\underset{\{{A,B}\}}{\arg\min} & \underset{g(z)}{\underset{︸}{\frac{1}{2}{{y - {Pz}}}_{2}^{2}}} \\{{subject}{to}} & {{z - {{vec}\left\{ {AB}^{T} \right\}}} = 0} \\ & {{0 \leq A},{B \leq 1},}\end{matrix}} & (2)\end{matrix}$where the matrix P∈

^(mns×mno) is the matrix form of operator

and the operator vec simply vectorizes a matrix into a single 1D vector(e.g. using column-major order as conducted by the software MATLABavailable from MathWorks of Natick, Mass.).

Then, the Augmented Lagrangian of this system is formulated as

$\begin{matrix}{{\mathcal{L}_{\rho}\left( {A,B,z,\xi} \right)} = {{g(z)} + {\xi^{T}\left( {{{vec}\left\{ {AB}^{T} \right\}} - z} \right)} + {\frac{\rho}{2}{{{\left\{ {AB}^{T} \right\} - z}}_{2}^{2}.}}}} & (3)\end{matrix}$In the scaled form, this Augmented Lagrangian is written as

$\begin{matrix}{{{\mathcal{L}_{\rho}\left( {A,B,z,u} \right)} = {\underset{g(z)}{\underset{︸}{\frac{1}{2}{{y - {Pz}}}_{2}^{2}}} + {\frac{\rho}{2}{{{{vec}\left\{ {AB}^{T} \right\}} - z + u}}_{2}^{2}}}},} & (4)\end{matrix}$where u=(1/ρ)ξ.

The ADMM algorithm then consists of three separate updates (or proximaloperators) that are iteratively executed as

$\begin{matrix}{\left. z\leftarrow{{\underset{\{ z\}}{\arg{}\min}\frac{1}{2}{{y - {Pz}}}^{\frac{2}{2}}} + {\frac{\rho}{2}{{z - v}}^{\frac{2}{2}}}} \right.,{v = {{AB}^{T} + u}}} & (5)\end{matrix}$ $\begin{matrix}{{\left\{ {A,B} \right\}\underset{\{{A,B}\}}{\leftarrow}{\arg\min{{V - {AB}^{T}}}_{F}^{2}}},{V = {{ivec}\left\{ {u - z} \right\}}}} & (6)\end{matrix}$ $\begin{matrix}\left. u\leftarrow{u + {AB}^{T} - z} \right. & (7)\end{matrix}$

Here, the operator ivec {⋅} reshapes a vector into a matrix and undoeswhat the operator vec does to vectorize a matrix. Equations 5-7 may besolved iteratively, each time using the latest output from the previousstep.

1.1 Efficient z-Update

Equation 5 is an unconstrained linear problem that may be re-written asa single linear equation system:

$\begin{matrix}{{\begin{bmatrix}P \\{\rho I}\end{bmatrix}z} = \begin{bmatrix}y \\{\rho v}\end{bmatrix}} & (8)\end{matrix}$

This system is large-scale, but all operations may be expressed asmatrix-free function handles so the matrices are never explicitlyformed. A variety of different solvers that may be used to solve thissystem for z. For example, the very simple simultaneous algebraicreconstruction technique (SART) of MATLAB may be utilized.

To increase computational efficiency, it would be desirable to derive aclosed-form solution for the z-update. This may facilitate a real-timeimplementation of the entire algorithm. One approach to deriving aclosed form solution starts with the normal equations for Equation 8:{circumflex over (z)}=(P ^(T) P+ρ ² I)⁻¹(P ^(T) y+ρ ² v)  (9)To find a closed form solution for this, the matrix inverse of(P^(T)P+ρp²I) is derived. Since P converts a light field into a focusstack and the Fourier Slice Theorem dictates that refocus in the primaldomain is a slicing in the Fourier domain, the closed form solution inthe frequency domain may be derived. Using this insight, one may write

$\begin{matrix}{{P^{T}P} = {{F_{4D}^{- 1}\left( {\sum\limits_{i = 1}^{s}{O_{i}^{*}O_{i}}} \right)}{F_{4D}.}}} & (10)\end{matrix}$Here, s is the number of slices in the focal stack, O_(i) is a diagonalmatrix representing the slicing operator for focal slice in the 4Dfrequency domain. F_(4D) and F_(4D) ⁻¹, represent the discrete 4DFourier transform and its inverse, respectively.

An expected algebraic expression for the matrix inverse is

$\begin{matrix}{\left( {{P^{T}P} + {\rho I}} \right)^{- 1} = {F_{4D}^{- 1}\frac{1}{{\sum\limits_{i = 1}^{s}{O_{i}^{*}O_{i}}} + {\rho^{2}I}}F_{4D}}} & (11)\end{matrix}$It will be appreciated that such a closed form solution may provide asolution more quickly than an iterative algorithm, since no iterationsare required. Nevertheless, if sufficient computational resources areavailable, then an iterative algorithm is also suitable.1.2 Efficient A,B-Update

The A, B-update (Eq. 6) is a nonnegative matrix factorization (NMF)problem. In this case, it is the easiest possible NMF problem. Standardsolutions for this and more advanced NMF approaches are detailed inSub-section 2 below.

1.3 Dealing with Color Channels

In the above derivations, a grayscale factorization was assumed or itwas assumed that each color channel may be treated independently. Insome cases, this may not provide a satisfactory approximation; forexample, two color SLMs may introduce color crosstalk, which is notmodeled above. In addition, in some embodiments, the display system mayuse a combination of a grayscale LCoS in the image plane and a color LEDor OLED array in the pupil plane. In this case, all color channels arelinked.

In accounting for linked color channels, neither the z-update nor theu-update change—these may be computed independently per color channel inevery ADMM iteration. However, the matrix factorization routineA,B-update does change. Instead of factoring each color channelA_(R,G,B)/B_(R,G,B) independently, a single factorization is performedfor all color channels of A simultaneously (B does not contain colorchannels in this case) as

$\begin{matrix}{\underset{\{{A,B}\}}{\arg\min}{{{\begin{pmatrix}V_{R} \\V_{G} \\V_{B}\end{pmatrix} - {\begin{pmatrix}A_{R} \\A_{G} \\A_{B}\end{pmatrix}B^{T}}}}_{F}}^{2}} & (12)\end{matrix}$2 Matrix Factorization Variants

Nonnegative matrix factorization is one approach to decompose a matrixinto a sum of nonnegative rank-one matrices. The decomposition problemis not convex, therefore solutions are not straightforward. The problemand possible solutions will now be discussed.

The problem may be stated as that of decomposing a matrix X into a sumof rank-one matrices

$\begin{matrix}{{{X \approx {\sum\limits_{k = 1}^{K}{a_{k} \circ b_{k}}}} = {AB}^{T}},} & (13)\end{matrix}$where X∈

^(M×N):x_(ij)≥0, A∈

^(M×K):a_(ik)≥0, and B∈

^(N×K):b_(jk)≥0. A sum of rank-one matrices results in a rank-Kapproximation of the original matrix.

A least squared error solution to the problem may be found by optimizingthe following objective function:

$\begin{matrix}\begin{matrix}\underset{\{{A,B}\}}{\arg\min} & {{J\left( {A,B} \right)} = {{\frac{1}{2}{{X - {AB}^{T}}}_{F}^{2}} = {\frac{1}{2}{\sum\limits_{ij}\left( {x_{ij} - \left( {AB}^{T} \right)_{ij}} \right)}}}} \\{{subject}{to}} & {A,{B \geq 0},}\end{matrix} & (14)\end{matrix}$where the Forbenius norm of a matrix is given as ∥X∥_(F) ²=Σ_(ij)x_(ij)².2.1 Alternating Lease Squares Approach

The cost function J(A, B)=½∥X−AB^(T)∥_(F) ² is both nonlinear andnon-convex, with a number of local minima. When fixing either A or B,solving for the other matrix is convex. Without considering thenonnegativity constraints, an alternating least squares approach,expected to converge, may be employed to solve the factorizationproblem. For this purpose, each factorization matrix is updated whilefixing the other in an alternating manner. The individual updates arecomputed using a gradient descent method:A←A−α _(A)∇_(A) J(A,B)B←B−α _(B)∇_(B) J(A,B)  (15)where ∇_(A,B) J (A, B) are the derivatives of the cost function withrespect to the individual factorization matrices, α_(A, B) theirrespective step lengths. As shown in the following subsection, oneapproach to choosing the step length is to pick them such that theupdate rules become multiplicative. Before discussing the step lengths,the gradients are considered and may be given as:∇a _(ik) J(A,B)=Σ_(u=1) ^(N)(x _(iu)−Σ_(l=1) ^(K)(a _(il) b _(ul)))(−b_(uk))∇b _(jk) J(A,B)=Σ_(v=1) ^(M)(x _(vj)−Σ_(l=1) ^(K)(a _(vl) b _(jl)))(−a_(vk))  (16)In matrix form, the gradients may be written as:∇_(A) J(A,B)=−(X−AB ^(T))B∇_(B) J(A,B)=−(A ^(T)(X−AB ^(T)))^(T).  (17)2.2 Multiplicative Update Rules

As noted above, a key to choosing the step length is that by combiningthem with the steepest descent direction, the additive update rules (Eq.15) may be written in a purely multiplicative way. Under the conditionsthat x_(ij)≥0 and A, B are initialized with positive values,multiplicative update rules provide that the factorization matricesremain positive throughout the iterative update process. The followingstep lengths result in multiplicative update rules:α_(A) =AØ((AB ^(T))B)α_(B) =BØ(A ^(T)(AB ^(T)))^(T)  (18)Combining Equations 15, 17, 18 yields

$\begin{matrix}{{\left. A\leftarrow{A - {\left( {A\left( {\left( {AB}^{T} \right)B} \right)} \right)\left( {{- \left( {X - {AB}^{T}} \right)}B} \right)}} \right. = {\left( {{A\left( {AB}^{T} \right)B} + {A{XB}} - {A\left( {AB}^{T} \right)B}} \right)\left( {\left( {AB}^{T} \right)B} \right)}}{\left. B\leftarrow{B - {\left( {B\left( {A^{T}\left( {AB}^{T} \right)}^{T} \right)} \right)\left( {- {A^{T}\left( {X - {AB}^{T}} \right)}} \right)^{T}}} \right. = {\left( {{B\left( {A^{T}\left( {AB}^{T} \right)}^{T} \right)} + {B\left( {A^{T}X} \right)^{T}} - {B\left( {A^{T}\left( {AB}^{T} \right)} \right)^{T}}} \right){\left( {A^{T}\left( {AB}^{T} \right)} \right)^{T}.}}}} & (19)\end{matrix}$The following multiplicative update rules are simplified versions ofEquation 19:A←A

(XB)Ø((AB ^(T))B),B←B

(A ^(T) X)^(T)Ø(A ^(T)(AB ^(T)))^(T).  (20)Starting from an initial guess that contains only positive values(usually random noise) and assuming that the data matrix X isnonnegative, these update rules are expected to keep A and B positivethroughout the iterative process. In practice, a small value is added tothe divisor so as to avoid division by zero.2.3 Weighted Nonnegative Matrix Factorization

$\begin{matrix}\begin{matrix}\underset{\{{A,B}\}}{\arg\min} & {{\frac{1}{2}{{X - {AB}^{T}}}_{W}^{2}} = {\frac{1}{2}{\sum\limits_{ij}{w_{ij}\left( {x_{ij} - \left( {AB}^{T} \right)_{ij}} \right)}^{2}}}} \\{{subject}{to}} & {A,{B \geq 0},}\end{matrix} & (21)\end{matrix}$

The multiplicative update rules may be modified to include weights foreach matrix element x_(ij)A←A

((W

X)B)Ø((W

(AB ^(T)))B),B←B

(A ^(T)(W

X))^(T)Ø(A ^(T)((AB ^(T))

W))^(T),  (22)where W is a weight matrix of the same size as X.2.4 Projected NMF

The projected NMF adds an additional projection matrix P to theobjective function that stays fixed throughout the optimizationprocedure:

$\begin{matrix}\begin{matrix}\underset{\{{A,B}\}}{\arg\min} & \begin{matrix}{\left( {A,B} \right) = \text{⁠}{{\frac{1}{2}{{X - {PAB}^{T}}}_{F}^{2}} =}} \\{J\frac{1}{2}{\sum\limits_{ij}\left( {x_{ij} - {\sum\limits_{i}{p_{li}\left( {\sum\limits_{k}{a_{ik}b_{jk}}} \right)}}} \right)^{2}}}\end{matrix} \\{{subject}{to}} & {A,{B \geq 0},}\end{matrix} & (23)\end{matrix}$Here, A and B remain unchanged to the previous subjections in theirdimensions but X∈

^(L×N): x_(lj)≥0 is in the space spanned by P∈

^(L×M). The gradients for this formulation are∇a _(ik) J(A,B)=Σ_(lj)(x _(lj)−Σ_(i) p _(li)Σ_(k) a _(ik) b _(jk))(−p_(li) b _(jk))∇b _(jk) J(A,B)=Σ_(l)(x _(lj) −E _(i) p _(li) a _(ik) b _(jk))(−Σ_(i) p_(li) a _(ik)),  (24)Which may be written in matrix form as∇_(A) J(A,B)=−P ^(T)(X−P(AB ^(T)))B∇_(B) J(A,B)=−A ^(T) P ^(T)(X−P(AB ^(T))).  (25)Choosing the step lengthα_(A) =AØ(P ^(T)(PAB ^(T))B)α_(B) =BØ((PA)^(T)(PAB ^(T)))^(T)  (26)leads to the following multiplicative update rules:A←A

(P ^(T) XB)Ø(P ^(T)(PAB ^(T)))^(T),B←B

((PA)^(T) X)^(T)Ø((PA)^(T)(PAB ^(T)))^(T),  (27)2.5 Projected Weighted NMF

For the projected NMF, weights may be added to the light field yieldingthe following update rulesA←A

(P ^(T)(W

X)B)Ø(P ^(T)(W

PAB ^(T))B),B←B

((PA)^(T)(W

X))^(T)Ø((PA)^(T)(W

PAB ^(T)))^(T).  (28)

It will be appreciated that each of the processes, methods, andalgorithms described herein and/or depicted in the figures may beembodied in, and fully or partially automated by, code modules executedby one or more physical computing systems, hardware computer processors,application-specific circuitry, and/or electronic hardware configured toexecute specific and particular computer instructions. For example,computing systems may include general purpose computers (e.g., servers)programmed with specific computer instructions or special purposecomputers, special purpose circuitry, and so forth. A code module may becompiled and linked into an executable program, installed in a dynamiclink library, or may be written in an interpreted programming language.In some embodiments, particular operations and methods may be performedby circuitry that is specific to a given function.

Further, certain embodiments of the functionality of the presentdisclosure are sufficiently mathematically, computationally, ortechnically complex that application-specific hardware or one or morephysical computing devices (utilizing appropriate specialized executableinstructions) may be necessary to perform the functionality, forexample, due to the volume or complexity of the calculations involved orto provide results substantially in real-time. For example, a video mayinclude many frames, with each frame having millions of pixels, andspecifically programmed computer hardware is necessary to process thevideo data to provide a desired image processing task or application ina commercially reasonable amount of time.

Code modules or any type of data may be stored on any type ofnon-transitory computer-readable medium, such as physical computerstorage including hard drives, solid state memory, random access memory(RAM), read only memory (ROM), optical disc, volatile or non-volatilestorage, combinations of the same and/or the like. In some embodiments,the non-transitory computer-readable medium may be part of one or moreof the local processing and data module (140), the remote processingmodule (150), and remote data repository (160). The methods and modules(or data) may also be transmitted as generated data signals (e.g., aspart of a carrier wave or other analog or digital propagated signal) ona variety of computer-readable transmission mediums, includingwireless-based and wired/cable-based mediums, and may take a variety offorms (e.g., as part of a single or multiplexed analog signal, or asmultiple discrete digital packets or frames). The results of thedisclosed processes or process steps may be stored, persistently orotherwise, in any type of non-transitory, tangible computer storage ormay be communicated via a computer-readable transmission medium.

Any processes, blocks, states, steps, or functionalities in flowdiagrams described herein and/or depicted in the attached figures shouldbe understood as potentially representing code modules, segments, orportions of code which include one or more executable instructions forimplementing specific functions (e.g., logical or arithmetical) or stepsin the process. The various processes, blocks, states, steps, orfunctionalities may be combined, rearranged, added to, deleted from,modified, or otherwise changed from the illustrative examples providedherein. In some embodiments, additional or different computing systemsor code modules may perform some or all of the functionalities describedherein. The methods and processes described herein are also not limitedto any particular sequence, and the blocks, steps, or states relatingthereto may be performed in other sequences that are appropriate, forexample, in serial, in parallel, or in some other manner. Tasks orevents may be added to or removed from the disclosed exampleembodiments. Moreover, the separation of various system components inthe embodiments described herein is for illustrative purposes and shouldnot be understood as requiring such separation in all embodiments. Itshould be understood that the described program components, methods, andsystems may generally be integrated together in a single computerproduct or packaged into multiple computer products.

In the foregoing specification, the invention has been described withreference to specific embodiments thereof. It will, however, be evidentthat various modifications and changes may be made thereto withoutdeparting from the broader spirit and scope of the invention. Thespecification and drawings are, accordingly, to be regarded in anillustrative rather than restrictive sense.

Indeed, it will be appreciated that the systems and methods of thedisclosure each have several innovative aspects, no single one of whichis solely responsible or required for the desirable attributes disclosedherein. The various features and processes described above may be usedindependently of one another, or may be combined in various ways. Allpossible combinations and subcombinations are intended to fall withinthe scope of this disclosure.

Certain features that are described in this specification in the contextof separate embodiments also may be implemented in combination in asingle embodiment. Conversely, various features that are described inthe context of a single embodiment also may be implemented in multipleembodiments separately or in any suitable subcombination. Moreover,although features may be described above as acting in certaincombinations and even initially claimed as such, one or more featuresfrom a claimed combination may in some cases be excised from thecombination, and the claimed combination may be directed to asubcombination or variation of a subcombination. No single feature orgroup of features is necessary or indispensable to each and everyembodiment.

It will be appreciated that conditional language used herein, such as,among others, “can,” “could,” “might,” “may,” “e.g.,” and the like,unless specifically stated otherwise, or otherwise understood within thecontext as used, is generally intended to convey that certainembodiments include, while other embodiments do not include, certainfeatures, elements and/or steps. Thus, such conditional language is notgenerally intended to imply that features, elements and/or steps are inany way required for one or more embodiments or that one or moreembodiments necessarily include logic for deciding, with or withoutauthor input or prompting, whether these features, elements and/or stepsare included or are to be performed in any particular embodiment. Theterms “comprising,” “including,” “having,” and the like are synonymousand are used inclusively, in an open-ended fashion, and do not excludeadditional elements, features, acts, operations, and so forth. Also, theterm “or” is used in its inclusive sense (and not in its exclusivesense) so that when used, for example, to connect a list of elements,the term “or” means one, some, or all of the elements in the list. Inaddition, the articles “a,” “an,” and “the” as used in this applicationand the appended claims are to be construed to mean “one or more” or “atleast one” unless specified otherwise. Similarly, while operations maybe depicted in the drawings in a particular order, it is to berecognized that such operations need not be performed in the particularorder shown or in sequential order, or that all illustrated operationsbe performed, to achieve desirable results. Further, the drawings mayschematically depict one more example processes in the form of aflowchart. However, other operations that are not depicted may beincorporated in the example methods and processes that are schematicallyillustrated. For example, one or more additional operations may beperformed before, after, simultaneously, or between any of theillustrated operations. Additionally, the operations may be rearrangedor reordered in other embodiments. In certain circumstances,multitasking and parallel processing may be advantageous. Moreover, theseparation of various system components in the embodiments describedabove should not be understood as requiring such separation in allembodiments, and it should be understood that the described programcomponents and systems may generally be integrated together in a singlesoftware product or packaged into multiple software products.Additionally, other embodiments are within the scope of the followingclaims. In some cases, the actions recited in the claims may beperformed in a different order and still achieve desirable results.

Accordingly, the claims are not intended to be limited to theembodiments shown herein, but are to be accorded the widest scopeconsistent with this disclosure, the principles and the novel featuresdisclosed herein.

We claim:
 1. A head-mounted display system comprising: a light sourcecomprising a plurality of spatially distinct light output locations; aspatial light modulator configured to modulate light from the lightsource; and projection optics configured to direct light from thespatial light modulator for propagation into an eye of a viewer, whereinthe display system is configured to display a virtual object on a depthplane by injecting a set of parallactically-disparate intra-pupil imagesof the virtual object into the eye, wherein injecting the set ofparallactically-disparate intra-pupil images comprises changing a pathof light from a light source to the spatial light modulator of thehead-mounted display by activating different ones of the light outputlocations to increase a magnitude of parallax disparity between theparallactically-disparate intra-pupil images as the virtual objectdecreases in perceived distance to the viewer.
 2. The display system ofclaim 1, configured to decrease a lateral separation between activatedlight output locations with decreases in the perceived distance of thevirtual object to the viewer.
 3. The display system of claim 1,configured to change the activated light output locations duringinjection of at least one of the intra-pupil images into the eye.
 4. Thedisplay system of claim 3, configured to change the activated lightoutput locations at a speed higher than an update rate of the parallaximage on the spatial light modulator.
 5. The display system of claim 1,configured to synchronize activation of different light output locationswith formation of different parallactically-disparate intra-pupil imagesby the spatial light modulator.
 6. The display system of claim 5,wherein the different parallactically-disparate intra-pupil images areformed at different locations across the spatial light modulator.
 7. Thedisplay system of claim 6, further comprising a lenslet array betweenthe spatial light modulator and the projection optics, wherein thelenslet array is configured to direct light from the different locationsof the spatial light modulator at different angles to the projectionoptics.
 8. The display system of claim 1, further comprising a prismbetween the spatial light modulator and the projection optics, whereinthe prism is configured to direct light from different locations of thespatial light modulator at different angles to the projection optics. 9.The display system of claim 1, wherein the light source, the spatiallight modulator, and the projection optics are configured to direct aplurality of beams of light to the eye of the viewer, wherein each beamof light corresponds to an associated parallactically-disparate image,and wherein each beam of light is collimated.
 10. The display system ofclaim 9, wherein each collimated beam of light has a diameter less thanabout 0.5 mm when incident on the eye.
 11. The display system of claim1, wherein the spatial light modulator is a digital light processing(DLP) chip.
 12. The display system of claim 1, wherein the projectionoptics directs the light to a waveguide comprising incoupling opticalelements and outcoupling optical elements, wherein the incouplingoptical elements are configured to incouple the light into the waveguideand the outcoupling optical elements are configured to output the lightfrom the waveguide into the eye of the viewer.
 13. The display system ofclaim 1, wherein the light source comprises a light-emitting diodearray.
 14. The display system of claim 13, wherein the light-emittingdiode array is a 2d array.
 15. The display system of claim 1, whereinthe light source comprises: a light emitter; and a spatial lightmodulator configured to selectively change a location of light receivedfrom the light emitter and propagating away from the spatial lightmodulator.
 16. The display system of claim 1, wherein the light sourcecomprises a fiber scanner.
 17. The display system of claim 1, whereininjecting the set of parallactically-disparate intra-pupil imagescomprises simultaneously injecting multiple ones of the intra-pupilimages.
 18. The display system of claim 17, wherein the set ofparallactically-disparate intra-pupil images consists of two or moreviews of the virtual object.
 19. The display system of claim 1,configured to inject the set of parallactically-disparate intra-pupilimages by temporally sequentially injecting theparallactically-disparate intra-pupil images into the eye of the viewerwithin a flicker fusion threshold.
 20. The display system of claim 19,wherein the flicker fusion threshold is 1/60 of a second.